Senior Data Engineer
About On The Spot
On The Spot Development is a software development company that builds R&D teams for heavily invested startups and tech companies from the UK, EU, and Israel.
This job opportunity is for a position within one of our teams. You will be directly employed by On The Spot and will collaborate closely with our partner companies, contributing to their product's journey.
About the team
As a part of Orca cloud security suite, we are developing a product that processes massive amounts of real-time data. Our team drives Cloud Detection and Response (CDR) offering, managing large-scale API log ingestion, comprehensive stream and batch log analysis, and high-performance real-time services. By connecting software engineering, big data processing, cloud infrastructure and security, we are delivering scalable and reliable systems.
About the role
We are looking for a highly skilled Senior Data Engineer to join our team and lead the development of scalable, efficient, and reliable data pipelines for big data processing. You will be responsible for designing, implementing, and optimizing data workflows, ensuring data quality, and enabling real-time security features. If you have a passion for large-scale data processing and a strong background in building ETL/ELT pipelines, we’d love to hear from you!
[01]
Responsibilities
- Design, develop, and maintain scalable and robust data pipelines for processing large datasets
- Optimize ETL/ELT workflows to ensure high performance, scalability, and efficiency
- Work with structured and unstructured data from multiple sources (e.g., logs, events, databases, APIs, and streams)
- Ensure data integrity, quality, and governance by implementing validation, monitoring, and logging mechanisms
- Implement real-time and batch processing frameworks using modern big data technologies
- Manage and optimize data storage solutions, including cloud-based data warehouses and lakehouses
- Automate data pipeline deployments using CI/CD practices and infrastructure-as-code
- Stay up-to-date with the latest trends and best practices in big data engineering
[01]
Required skills & experience
- 5+ years of experience in designing and developing data pipelines for big data processing
- Expertise in Python, Scala, or Java for data engineering tasks
- Strong experience with ETL/ELT frameworks such as Apache Airflow, dbt, or Luigi
- Proficiency with big data technologies like Apache Spark, Flink, Kafka, or Hadoop
- Experience working with cloud platforms such as AWS, GCP, or Azure (e.g., S3, Athena, Redshift, BigQuery, Databricks)
- Solid understanding of SQL, NoSQL databases, and data modeling techniques
- Hands-on experience with data lakehouses and data warehouses (Iceberg, Delta Lake, Snowflake, or similar)
- Experience with stream processing frameworks (Kafka Streams, Apache Flink, or Spark Streaming)
- Familiarity with containerization and orchestration (Docker, Kubernetes) is a plus
- Knowledge of CI/CD pipelines, infrastructure as code (Terraform, CloudFormation) is desirable
[01]
Preferred qualifications
- Experience with cloud security products
- Experience with best practices for data handling
- Background in real-time anomaly detection and event-driven architectures
[02]
[03]
Benefits
- Work in a highly professional team. Informal and friendly atmosphere in the team
- Paid vacation — 20 business days per year, 100% sick leave payment
- Equipment provision
- Partially compensated educational costs (for courses, certifications, professional events, etc.)
- Legal and Accounting support in Poland
- Ability to work from our comfortable office in Warsaw at Prosta 51
- 5 sick days per year
- Medical insurance (after the end of the probationary period)
- Flexible working hours – we care about you (!) and your output
- English and Polish classes 2 times a week (online)
- Bright and memorable corporate life: corporate parties, gifts to employees on significant dates
Join us
If you'd like to work from Poland, having Polish citizenship, karta pobytu, or D-type work visa for Poland is obligatory to be considered for this position. Thank you!
.png)
Contribute to our growth
Know someone perfect for this role? Let us know about them. We have a referral program to recognize your support.
.png)
.svg)