Sandipan Ghosh – Medium

Sandipan Ghosh

HOW TO PARSE XML DATA TO A SAPRK DATAFRAME

Purpose

2 min readMar 14, 2022

--

HOW TO PARSE XML DATA TO A SAPRK DATAFRAME

--

Sandipan Ghosh

Data Streaming — AWS Kinesis

In my current organisation, we had implemented Kafka from end to end, from acquiring resources, installing Kafka, developing the producer…

14 min readOct 16, 2021

--

Data Streaming — AWS Kinesis

--

Sandipan Ghosh

Cloud Computations — Quick data analysis with AWS Athena, Glue and Databricks spark

Throughout my carrier, I always had a situation that I had to fix failing production jobs. Most of the time, the debug involved analysis…

6 min readOct 3, 2021

--

Cloud Computations — Quick data analysis with AWS Athena, Glue and Databricks spark

--

Sandipan Ghosh

Spark — Write single file per (hive) partitions

I have faced the problem of a small file in Hadoop too many times during the spark ETL process which is writing data to partitioned Hive…

6 min readJun 25, 2021

--

Spark — Write single file per (hive) partitions

--

Sandipan Ghosh

How to Install Spark 3 on Windows 10

I have been using spark for a long time. It is an excellent, distributed computation framework. I use this regularly at work, and I also…

4 min readMar 10, 2021

--

How to Install Spark 3 on Windows 10

--

Sandipan Ghosh

How to download really big data sets for big data testing

For a long time, I have been working with big data technologies, like MapReduce, Spark, Hive, and very recently I have started working on…

3 min readAug 3, 2020

--

How to download really big data sets for big data testing

--

Sandipan Ghosh

Sandipan Ghosh

Bigdata solution architect & Lead data engg with experience in building data-intensive applications,tackling challenging architectural and scalability problems.

Following

Help
Status
About
Careers
Blog
Privacy
Terms
Text to speech
Teams