Interview Questions for Big Data Processing (e.g., Hadoop, Spark)

Hadoop and Spark are both powerful frameworks for big data processing, but they differ significantly in their architecture and approach. Think of Hadoop as a sturdy, reliable workhorse, best suited for batch processing of massive datasets. Spark, on the other hand, is a nimble racehorse, excelling at both batch and real-time processing with significantly improved speed.

Hadoop relies on the MapReduce paradigm, which involves breaking down a large task into smaller, independent map and reduce operations. This is inherently slower due to its disk-based processing. Spark, conversely, uses in-memory computation. This means data is stored in RAM, drastically reducing the time it takes to process information. Imagine trying to sort a massive pile of papers: Hadoop would meticulously move each paper to its correct location on a table, while Spark would keep all the papers on the table, shuffling them around in memory for much faster sorting.

Another key difference lies in their data management. Hadoop’s HDFS (Hadoop Distributed File System) is designed for storing massive, immutable datasets. Spark supports a variety of data sources and can interact with many data formats, offering greater flexibility.

The Hadoop Distributed File System (HDFS) is a crucial component of Hadoop, designed to store and process vast amounts of data across a cluster of machines. It’s composed of three main components:

• NameNode: The master node, responsible for managing the file system metadata (e.g., file locations, permissions). Think of it as a librarian cataloging all the books in the library.
• DataNodes: The worker nodes, storing actual data blocks. Each DataNode holds a portion of the overall data, distributed across the cluster. These are like the bookshelves holding the actual books.
• Secondary NameNode: A helper node that periodically creates a checkpoint of the NameNode’s metadata, preventing data loss in case of NameNode failure. It’s like a backup system for the librarian’s catalog.

HDFS’s design emphasizes high throughput and fault tolerance. Data is replicated across multiple DataNodes, ensuring availability even if some nodes fail. This redundancy is crucial for managing the potential failures inherent in large-scale distributed systems.

Hadoop offers diverse data storage options based on the needs of the processing task. The primary data storage is the Hadoop Distributed File System (HDFS), as discussed earlier. However, Hadoop also integrates with other storage systems:

• HDFS: Designed for large-scale, batch processing, storing data in a distributed manner across multiple nodes.
• Hadoop YARN (Yet Another Resource Negotiator): This is not strictly a storage system, but rather a resource manager that manages cluster resources and allows applications like MapReduce and Spark to run. It indirectly interacts with storage via the execution of processing jobs.
• Hive: Provides a SQL-like interface for querying data stored in HDFS. It makes it easier for users familiar with SQL to work with big data.
• HBase: A NoSQL database built on top of HDFS, offering column-oriented storage for fast read/write operations on large datasets. Ideal for applications requiring low-latency access to data.

The choice of storage depends on the specific application requirements. For example, HBase is suitable for real-time analytics, while HDFS is better for large-scale batch processing.

MapReduce is a programming model and processing framework that forms the core of Hadoop’s batch processing capabilities. It’s a powerful way to parallelize computation over large datasets. Think of it as a sophisticated assembly line for data processing.

It works in two stages:

• Map: This stage involves breaking down a large dataset into smaller, independent chunks. Each chunk is then processed individually by a mapper function, producing intermediate key-value pairs. This is like sorting the individual components of the product on the assembly line.
• Reduce: The intermediate key-value pairs generated by the map stage are then grouped and aggregated by a reducer function. This stage produces the final results. This is like assembling the final product from its sorted components.

For example, to count the word occurrences in a large text file, the mapper would split the file, count words in each chunk, and produce key-value pairs (word, count). The reducer would then combine the counts for each word to get the total count.

Hadoop, while powerful, has both advantages and disadvantages:

Advantages:

• Scalability: Handles massive datasets easily by distributing them across a cluster of machines.
• Fault Tolerance: Data replication ensures high availability even with node failures.
• Cost-Effective: Utilizes commodity hardware, making it relatively inexpensive to set up compared to proprietary solutions.
• Mature Ecosystem: A large and active community provides ample support, tools, and resources.

Disadvantages:

• Complex Setup and Configuration: Requires significant expertise to set up and manage properly.
• Slow Performance for certain tasks: MapReduce’s disk-based processing is slow for iterative algorithms and real-time applications.
• Limited real-time capabilities: It is not optimal for tasks requiring immediate responses.
• High latency: Data retrieval from HDFS can be slow due to disk access.

Choosing Hadoop depends on the specific needs of the project. Its strengths lie in managing massive datasets for batch processing where performance is less critical than scalability and cost-effectiveness.

In Spark, Resilient Distributed Datasets (RDDs) are fundamental data structures that represent a collection of elements partitioned across a cluster. There are two main types of RDDs:

• Parallel Collections: These are created from existing collections in the driver program (e.g., lists, arrays). They are easily parallelized for processing.
• Hadoop Datasets: These are created from external data sources, such as HDFS files, or other databases. This is how Spark interacts with Hadoop’s data.

Both types provide fault tolerance through lineage tracking. If a partition of an RDD is lost, it can be recomputed from the previous operations that created it. This makes Spark extremely resilient to failures.

Furthermore, RDDs can be further categorized based on their persistence level (how long the data stays in memory):

• persist(): The RDD is stored in memory in a chosen format, improving subsequent access times.
• cache(): A shortcut for persist() with the default storage level.
• unpersist(): Removes the RDD from memory, freeing up space.

The choice of persistence level is important for balancing performance and resource consumption.

Spark’s DAG (Directed Acyclic Graph) scheduler is a crucial component responsible for efficiently scheduling and executing Spark jobs. It optimizes the execution of a series of transformations and actions on RDDs. Think of it as a sophisticated traffic controller for data processing.

A DAG represents the dependencies between transformations applied to RDDs. The scheduler analyzes this DAG, identifying stages based on the dependencies and data shuffling required. It then schedules these stages across the cluster, optimizing resource utilization and minimizing data movement.

The DAG scheduler’s key function is to:

• Analyze the DAG: Breaks down the job into stages, based on data dependencies.
• Stage scheduling: Creates tasks within each stage to be executed on the worker nodes.
• Resource allocation: Assigns resources (CPU, memory) to the tasks based on their needs.
• Data locality: Tries to schedule tasks on the worker nodes where the data is already located. Minimizing data transfer is a significant optimization.
• Fault tolerance: Handles failures by rescheduling tasks that fail.

This intelligent scheduling ensures that Spark jobs are executed efficiently, reducing processing time and improving overall performance.

Spark’s fault tolerance is a cornerstone of its reliability, ensuring that your application continues to run even if nodes in your cluster fail. It achieves this primarily through lineage tracking and data replication. Lineage tracking means Spark keeps track of all the transformations applied to your data. If a task fails, it can reconstruct the lost data by re-executing the transformations from the last checkpoint. Data replication involves storing multiple copies of your data across different nodes. If one node fails, the data is still accessible from other nodes.

Think of it like a recipe: Lineage is like having the instructions, and data replication is like having multiple copies of ingredients. If you lose some ingredients, you can still make the dish from the recipe and the remaining ingredients.

This resilience is crucial in large-scale data processing where hardware failures are common. It means you don’t need to restart your entire job from scratch if a single node goes down, significantly reducing downtime and improving efficiency.

Deploying a Spark application involves choosing the right execution environment. The most common methods include:

• Local Mode: Ideal for testing and development. The application runs on a single machine, utilizing all available cores. This is simple to set up but doesn’t leverage the power of a distributed cluster.
• Standalone Mode: A self-contained cluster managed directly by Spark. You have complete control over the cluster but require manual configuration and maintenance.
• Yarn (Yet Another Resource Negotiator): A resource manager in Hadoop that manages cluster resources. Spark runs as an application within the Yarn framework, allowing for efficient resource allocation across multiple applications.
• Mesos: A cluster manager that can manage resources from diverse environments. Spark can integrate with Mesos to leverage heterogeneous clusters.
• Kubernetes: A container orchestration system. Spark applications can be deployed as containers within a Kubernetes cluster, providing benefits like scalability, fault tolerance, and automated management.

The choice of deployment method depends on your specific needs, including the scale of your data, the complexity of your application, and the existing infrastructure.

Spark Streaming and Apache Kafka are both used for real-time data processing, but they serve different purposes and have distinct strengths.

Spark Streaming is a framework built on top of Spark Core that processes continuous streams of data. It divides the stream into mini-batches, applying Spark transformations to each batch. It’s excellent for handling high-volume, structured data streams and offers the benefits of Spark’s fault tolerance and in-memory computation. However, it might introduce some latency due to batching.

Apache Kafka is a distributed, fault-tolerant, high-throughput streaming platform that acts as a message broker. It is primarily responsible for storing and distributing data streams efficiently. It’s a robust messaging system capable of handling extremely high volumes of data with low latency. It doesn’t provide any processing capabilities itself but can feed data to various consumers, including Spark Streaming.

In essence, Kafka is the pipeline, and Spark Streaming is one of the tools you can use to process the data flowing through that pipeline. They often work together: Kafka stores and distributes the data stream, and Spark Streaming processes it.

Data partitioning in Spark involves dividing your dataset into smaller, manageable parts, distributing them across the cluster’s nodes to process in parallel. This is essential for performance optimization and efficient computation.

There are several ways to partition data:

• Hash Partitioning: Data is partitioned based on a hash function applied to a key field. This ensures relatively even data distribution.
• Range Partitioning: Data is partitioned based on ranges of values in a key field. This is suitable when your data is naturally ordered.
• Custom Partitioning: You can define your own partitioning logic based on specific business needs or data characteristics.

Effective partitioning leads to improved performance by reducing the amount of data each node needs to process, enhancing parallelism, and allowing for more efficient shuffling during operations like joins.

For example, if you’re analyzing user activity, you might partition data by user ID. This ensures that all activities for a particular user are processed by the same node.

Optimizing Spark performance is a multifaceted task. Some crucial strategies include:

• Data Partitioning: Choose the right partitioning strategy based on your data and processing requirements. Proper partitioning minimizes data shuffling and improves parallelism.
• Data Serialization: Use efficient serialization formats like Kryo instead of Java serialization to reduce data size and improve network transfer speed.
• Caching: Cache frequently accessed data in memory to reduce disk I/O and improve processing speed. Use persist() or cache() methods in your code.
• Broadcast Variables: Use broadcast variables to distribute small datasets to all nodes, avoiding repeated reads from storage.
• Join Optimization: Choose appropriate join types (e.g., broadcast hash join for small datasets) based on your data size and distribution.
• Code Optimization: Write efficient Spark code by using optimized transformations and avoiding unnecessary operations.
• Resource Allocation: Allocate sufficient resources (CPU, memory) to your Spark application based on the scale of your data.
• Tuning Configuration Parameters: Experiment with Spark configuration parameters to find optimal settings for your specific environment.

A systematic approach, involving profiling, analyzing performance metrics, and targeted optimizations, is crucial for significant performance gains.

Common performance bottlenecks in Spark applications often stem from:

• Data Skew: Uneven data distribution across partitions leading to some nodes processing significantly more data than others, causing delays.
• Network I/O: Inefficient data transfer between nodes due to poor partitioning, excessive shuffling, or slow network connections.
• Disk I/O: Frequent disk reads due to insufficient caching or slow storage systems. This can significantly slow down processing.
• Shuffle Operations: Inefficient shuffle operations (like during joins) can be major bottlenecks due to the substantial data movement involved.
• Garbage Collection: Frequent and lengthy garbage collection pauses can significantly impact performance.
• Insufficient Resources: Not allocating enough CPU, memory, or executors can lead to resource contention and slowdowns.

Identifying and addressing these bottlenecks often involves careful performance profiling and analysis using tools provided by Spark or external monitoring systems.

I have extensive experience with both Hive and Pig, two prominent tools used for processing large datasets in Hadoop ecosystems. While both are used for data warehousing and ETL (Extract, Transform, Load) processes, they differ significantly in their approach.

Hive provides a SQL-like interface for querying data stored in Hadoop. This makes it relatively easier for users familiar with SQL to interact with large datasets. It leverages the power of Hadoop’s distributed processing capabilities, allowing complex queries to be executed in parallel.

Pig, on the other hand, uses a higher-level scripting language (Pig Latin) that allows for more flexible data manipulation and transformation. It’s particularly useful when dealing with complex data transformations that are difficult to express using SQL. Pig offers a more procedural approach compared to Hive’s declarative style.

In a project involving customer transaction data, I used Hive to perform aggregations and generate reports on sales trends, leveraging its SQL-like interface for efficient query construction. For a separate project involving complex data cleaning and transformations of unstructured log data, I employed Pig Latin’s flexibility to handle various data cleaning tasks, enabling efficient data preparation for downstream analyses.

The choice between Hive and Pig depends on the complexity of the task and the user’s familiarity with SQL versus scripting languages. For straightforward data querying and reporting, Hive is often preferred. For more complex, iterative, and ad-hoc data transformations, Pig’s flexibility can be advantageous.

Schema-on-read and schema-on-write are two fundamental approaches to handling data schemas in big data processing. They differ primarily in when the schema is defined and enforced.

Schema-on-write means the data’s structure is defined before it’s written to storage. Think of it like filling out a pre-printed form – you must adhere to the existing fields and data types. This approach ensures data consistency and simplifies querying, as the schema is known upfront. However, it can be inflexible, requiring schema changes to accommodate evolving data structures. Examples include databases like traditional relational databases (e.g., MySQL, PostgreSQL) and some NoSQL databases with strict schema enforcement.

Schema-on-read, on the other hand, defines the schema during the reading or querying process. Imagine a blank sheet of paper – you can write whatever you want, and the structure is determined later when you analyze the information. This is highly flexible, accommodating evolving data structures and various data types within the same dataset. However, it can be less efficient for querying since the schema needs to be inferred during each read operation. This approach is common with NoSQL databases like MongoDB and data lake technologies where data is stored in a raw format (like JSON or Avro) without a predefined structure. Apache Hive is a classic example that uses Schema-on-Read on data stored in Hadoop.

In a nutshell: Schema-on-write prioritizes consistency and efficiency of querying; Schema-on-read prioritizes flexibility and accommodates evolving data.

Big Data processing uses several data formats, each with its strengths and weaknesses. The choice depends on factors such as query performance, data size, schema evolution, and storage efficiency.

• CSV (Comma Separated Values): A simple, human-readable format, widely used for data exchange. However, it lacks schema information and can be inefficient for large datasets.
• JSON (JavaScript Object Notation): A flexible, self-describing format commonly used for semi-structured data. It’s human-readable but can be less efficient than columnar formats for analytical processing.
• Parquet: A columnar storage format offering significant performance advantages for analytical queries. It supports schema evolution, compression, and efficient data filtering.
• ORC (Optimized Row Columnar): Another columnar format similar to Parquet, offering good performance and compression. It’s often compared to Parquet, and the best choice depends on specific needs and the Big Data processing system used.
• Avro: A row-oriented format that’s schema-aware, supporting schema evolution and efficient serialization. It’s particularly useful when dealing with complex data structures.

I have extensive experience with Avro, Parquet, and ORC, having used them in numerous big data projects. My experience highlights their distinct strengths:

Avro: I’ve used Avro when schema evolution was a key requirement. Its ability to handle schema changes without data loss is invaluable in dynamic environments. For example, in a project tracking customer interactions, Avro allowed us to add new fields (e.g., social media interactions) without needing to rewrite existing data. The schema evolution feature is managed through schema IDs, providing versioning.

Parquet: In projects focused on analytical processing, Parquet has consistently provided excellent performance. Its columnar storage enables efficient querying of specific columns, avoiding unnecessary data reads. I recall a project involving terabytes of sensor data where using Parquet reduced query times by several orders of magnitude compared to using CSV. The data compression capabilities are also very helpful.

ORC: I’ve found ORC to be a strong alternative to Parquet, especially in cases where data needs to be accessed randomly. While both are columnar, ORC’s nested structures and efficient handling of complex data structures often prove advantageous. In one project, ORC’s superior handling of nested JSON-like data significantly improved performance over Parquet in certain query scenarios.

The choice between these formats often comes down to specific project requirements and the underlying processing framework. For instance, Spark integrates seamlessly with all three, providing optimized readers and writers.

Data cleaning and preprocessing are crucial steps in any big data project. My approach involves a multi-stage process:

1. Data Discovery and Profiling: This involves understanding the data’s structure, identifying data types, and detecting potential issues like inconsistencies or missing values. Tools like Apache Spark’s DataFrame API or dedicated profiling tools are invaluable here.
2. Data Cleaning: This stage focuses on addressing issues like missing values (using imputation techniques or removing rows/columns), handling outliers (through capping, winsorization or removal), and correcting inconsistencies (data standardization and normalization). Regular expressions are frequently used for data cleaning.
3. Data Transformation: This involves transforming data into a suitable format for analysis. This might involve changing data types, creating new features, or aggregating data. Spark SQL and PySpark are commonly used for these transformations.
4. Data Validation: After cleaning and transforming, data validation ensures the quality and consistency of the processed data. This could involve checks for data type consistency, range checks, and uniqueness checks.

For example, in a project processing social media data, I used Spark to clean and process tweets, removing irrelevant characters, handling missing hashtags, and transforming textual data into numerical representations (e.g., using TF-IDF or word embeddings) for further analysis.

Handling missing data is a critical aspect of big data processing. The best approach depends on the nature of the missing data (missing completely at random, missing at random, or missing not at random), the amount of missing data, and the impact on the analysis.

• Deletion: Removing rows or columns with missing values is the simplest approach but can lead to significant data loss, especially if missing data is not random.
• Imputation: Replacing missing values with estimated values. Common techniques include mean/median/mode imputation, k-nearest neighbor imputation, and model-based imputation. For example, using Spark’s fillna function.
• Prediction: Predicting missing values using machine learning models. This is more complex but can provide more accurate estimates, especially for non-random missing data.

The choice of method should be driven by data characteristics and the downstream analysis. For example, in a project predicting customer churn, I used k-NN imputation for missing customer demographics and a machine learning model to predict missing purchase history values to avoid significant data loss.

Data transformation in big data involves converting raw data into a format suitable for analysis. Common techniques include:

• Aggregation: Combining multiple data points into summary statistics (e.g., SUM, AVG, COUNT).
• Filtering: Selecting specific subsets of data based on conditions.
• Joining: Combining data from multiple tables based on common keys.
• Pivoting: Restructuring data from rows to columns or vice versa.
• Feature Engineering: Creating new features from existing ones to improve model performance. This can involve using domain knowledge or applying mathematical/statistical functions.
• Data Type Conversion: Changing the data type of a column (e.g., string to integer).

For example, in a project analyzing website traffic, I used Spark SQL to aggregate page views by day, filter out bot traffic, and join the data with user demographics to create a dataset suitable for analyzing user behavior patterns.

The choice of transformations depends heavily on the analytic goal. For instance, generating time series features, applying one-hot encoding for categorical data, or using log transformations for skewed data are all example data transformations applied with the analytical goals in mind.

I have substantial experience with data warehousing concepts and technologies, having designed and implemented several data warehouses using various technologies. My experience encompasses the entire data warehousing lifecycle:

• Data Modeling: Defining the data model, choosing between dimensional modeling (star schema, snowflake schema) and other approaches based on the business requirements. This often involves creating ER diagrams and conceptual data models.
• ETL (Extract, Transform, Load): Designing and implementing ETL processes to extract data from various sources, transform it according to the data model, and load it into the data warehouse. I have experience with various ETL tools, including Apache Sqoop, Apache Kafka, and custom Spark-based ETL pipelines.
• Data Warehousing Technologies: I’ve worked with various data warehouse technologies, including Hadoop Distributed File System (HDFS), cloud-based data warehouses (e.g., AWS Redshift, Google BigQuery, Snowflake), and traditional relational databases optimized for data warehousing.
• Querying and Reporting: Developing efficient queries using SQL to retrieve data from the data warehouse and generating reports for business users.

For instance, in one project, I designed a data warehouse for a retail company using a star schema, extracting transactional data from multiple sources, performing data cleaning and transformation using Spark, and loading the data into an AWS Redshift cluster. This enabled the company to gain valuable insights into customer behavior and sales trends.

Data visualization is crucial for understanding complex Big Data insights. My experience spans several tools, each suited to different needs. I’m proficient in Tableau, known for its intuitive drag-and-drop interface and powerful analytical capabilities, ideal for creating interactive dashboards and reports for business stakeholders. I’ve also used Power BI extensively, leveraging its strong integration with Microsoft products and its capacity for creating compelling visualizations from diverse data sources. For more programmatic visualization, I’ve utilized libraries like Matplotlib and Seaborn in Python, offering fine-grained control over plot aesthetics and enabling the creation of publication-quality figures. In one project, I used Tableau to visualize customer churn patterns, revealing key demographic and behavioral factors contributing to churn, leading to targeted retention strategies. Another project involved using Matplotlib to visualize the performance of a Spark application over time, identifying bottlenecks and areas for optimization.

Data security and privacy are paramount in Big Data. My approach is multi-layered and addresses data at rest, in transit, and in use. For data at rest, encryption (both at the file and database level) is essential, using tools like AES-256. Data in transit requires secure protocols such as HTTPS and TLS. Access control mechanisms, using tools like Apache Ranger and Kerberos, are implemented to restrict access to sensitive data based on roles and permissions. Data anonymization and pseudonymization techniques are used to protect user identities. I also focus on data governance, defining clear data ownership and responsibility, adhering to regulations like GDPR and CCPA. Regular security audits and penetration testing are crucial to identify and address vulnerabilities. For example, in a project involving sensitive health data, we implemented differential privacy techniques to ensure privacy while still deriving meaningful insights.

I have substantial experience with AWS, Azure, and GCP for Big Data processing. AWS offers a comprehensive suite of services, including EMR (Elastic MapReduce) for Hadoop and Spark clusters, S3 for storage, and Redshift for data warehousing. I’ve utilized these services to build highly scalable and cost-effective solutions. Azure’s HDInsight provides similar capabilities, and I’ve found its integration with other Azure services, like Azure Data Lake Storage, particularly valuable. GCP’s Dataproc offers a managed Spark and Hadoop service, and its BigQuery is a powerful serverless data warehouse. Each platform has its strengths; the choice depends on existing infrastructure, budget, and specific requirements. For example, a project involving real-time data streaming favored Azure’s Event Hubs and Stream Analytics, while another project benefited from GCP’s BigQuery’s advanced SQL capabilities for complex analytical queries.

Monitoring and troubleshooting Big Data applications is crucial for performance and reliability. I use a combination of tools and techniques. For monitoring, I leverage platform-specific tools like CloudWatch (AWS), Azure Monitor, and Stackdriver (GCP), which provide metrics on resource utilization, job completion times, and error rates. Open-source tools like Prometheus and Grafana allow for customizable dashboards and alerts. For troubleshooting, I use application logs, system logs, and performance profiling tools to identify bottlenecks and errors. Techniques like analyzing Spark UI metrics, using YARN resource managers, and examining Hadoop logs are essential. Debugging often involves iterative analysis of logs, resource usage, and code. For instance, when a Spark job was running excessively long, I used the Spark UI to pinpoint a stage with high data skew, then optimized the data partitioning to resolve the issue.

NoSQL databases are essential for handling the variety and volume of data in Big Data environments. I have experience with various NoSQL databases, including MongoDB (document database), Cassandra (wide-column store), and HBase (column-oriented). MongoDB’s flexibility makes it suitable for semi-structured data, while Cassandra’s high availability and scalability are excellent for high-volume write operations. HBase’s performance on large datasets is ideal for certain analytical tasks. The choice of database depends on the specific data model and application requirements. For instance, a project involving social media data leveraged MongoDB’s flexibility to handle the varied structure of user data, whereas a project tracking sensor readings benefited from Cassandra’s high-throughput capabilities.

Data lineage refers to the complete history of a dataset, tracing its origins, transformations, and usage. In Big Data, understanding data lineage is critical for data quality, compliance, and troubleshooting. It answers the question: “Where did this data come from, how was it processed, and who used it?” Tools like Apache Atlas and open-source solutions help track data lineage by recording metadata about data transformations, sources, and destinations. A robust data lineage system is essential for auditing, debugging, and regulatory compliance. For example, if a data quality issue is identified, tracing its lineage can pinpoint the source of the problem and allow for corrective action. Similarly, data lineage is essential for demonstrating compliance with regulations requiring traceability of data processing.

One challenging project involved processing terabytes of real-time sensor data from a network of smart city devices. The challenge was to ingest, process, and analyze this data in real-time to identify anomalies and trigger alerts for traffic congestion, environmental pollution, and infrastructure failures. We initially faced performance bottlenecks due to data ingestion and processing limitations. My approach involved a multi-pronged strategy: first, we optimized the data ingestion pipeline by implementing Kafka as a distributed message queue, buffering the incoming data stream and allowing for parallel processing. Second, we used Spark Streaming to perform real-time processing, leveraging its fault-tolerance and scalability. Third, we optimized the Spark application by tuning the parameters, adjusting data partitioning, and employing broadcast variables. This resulted in a significant improvement in processing speed and reduced latency, enabling near real-time anomaly detection and improved city management.