बजट: USD 9,000-16,000
पोस्ट का समय:
AdTech company processing 2 billion plus events daily, growing 30% quarterly. Our Spark jobs are getting delayed during peak hours and data quality is suffering from schema evolution issues. Need help redesigning our ingestion pipeline, possibly moving to Kafka plus Flink, optimizing existing Spark jobs, and building a proper data quality framework. Stack is AWS with EMR, S3, Redshift, Airflow. Languages are Python and Scala. Starting with 2 months to fix critical issues, with potential extension for longer optimization work.