Senior Python Streaming & Data Engineer (RabbitMQ / Snowflake / PostgreSQL)
- Employment type
- Contract
- Workload
- 100%
- Location
- Bern
- Languages
- German, German
- First posted
- IT - Software Engineering
- Bern
- Contracting
- Part-time
- Snowflake, PostgreSQL, RabbitMQ
The two engineers, who together are responsible for the entire data processing chain, should ideally complement each other with their technical focuses as follows:
- Streaming & Performance: Focus on RabbitMQ consumer development, Python Concurrency, performance optimization, and stable processing under high load
- Data Integration & Recovery: Focus on Snowflake/PostgreSQL, reliable data ingestion as well as recovery, replay, and backfill processes
For both focuses, practical experience with RabbitMQ and demonstrable project experience in production environments are required.
Tasks:
- Development and optimization of a scalable Python application for processing RabbitMQ data streams
- Implementation and further development of RabbitMQ consumers for production streaming environments
- Integration and configuration of data ingestion into Snowflake and Snowflake Postgres
- Ensuring reliable message processing including acknowledgements, redelivery, backpressure, and deduplication
- Conception and implementation of robust recovery, replay, and backfill processes
- Optimization of latency, throughput, as well as CPU and memory consumption
- Analysis and resolution of performance bottlenecks, load peaks, and error cases
- Ensuring data completeness and data consistency for live and historical data
- Integration of reload processes using Kestra and dlt (dltHub) in cooperation with the internal Data Ingestion Engineer
- Development of automated tests as well as conducting performance and load tests
- Integration of the solution into Azure DevOps CI/CD and implementation of basic monitoring functions
- Close cooperation with internal platform and engineering teams as well as documentation and knowledge transfer
Profile:
- Very good practical experience in Python development, especially with long-running production applications
- Sound experience with RabbitMQ, especially consumer development, message processing, and error handling
- Practical experience with streaming systems under real load as well as their performance optimization
- Good knowledge of Snowflake and PostgreSQL, especially in the area of data ingestion and data integration
- Experience with recovery, replay, backfill, deduplication, and data consistency
- Very good understanding of acknowledgements, redelivery, backpressure, and message queuing
- Knowledge of Python Concurrency, CPU/memory profiling, and performance optimization
- Experience in the development and operation of production-ready data pipelines
- Knowledge of automated tests, CI/CD, and monitoring
- Experience with Azure DevOps as well as modern development and deployment processes
- Knowledge of Azure, AKS, Kestra, dlt, Kafka, or Debezium is an advantage
- Independent, structured, and solution-oriented way of working as well as strong teamwork skills
- Experience in cooperating with internal platform, development, and operations teams
Automatically translated from the original.
Posted 6 days ago