Hassle free and Quick way of loading an csv file into Azure SQL Database

Image for post
Image for post
Photo by Boitumelo Phetla on Unsplash

Over the weekend, I wanted to do a quick proof of concept on certain capabilities of Databricks and I wanted to use Azure SQL as a source. I faced quite bit of challenges and google was not kind enough to provide me a solution. I tried everything from bcp to bulk insert the file on my local computer and somehow it came out with errors which failed to fix.

Finally I managed to load the csv file into database and thought of sharing this with everyone, so that if you have to quickly load data into Azure SQL Database and…


Data Privacy

GDPR principles directly affect the storage, processing, and use of personal data in your Data Analytics platform.

General Data Protection Rules or as they call it GDPR across the world is good enough to scare anyone collecting personal data of its customers for their business. However, it is way beyond the famous “Right to be forgotten” statement which everyone has in their mind the moment they hear the word GDPR.

This regulation lays down rules relating to the protection of natural persons with regard to the processing of personal data and rules relating to the free movement of personal data.

As per the definition provided by the GDPR regulation

‘personal data’ means any information relating to an…


Image for post
Image for post
Photo by Sven Mieke on Unsplash

Yes, you heard it right. We were not getting over with the Infrastructure as code and now we are already talking about Diagram as a Code.

As an Architect, I have always believed that we should be able to write code too, so as to create a prototype of my designs and prove that what we preach does work. I jokingly call myself an “Architect who can code”

My take on the concept of a Diagram as a code? It would be extremely useful and fabulous to simply write a few lines of code that generate a Cloud Architecture Diagram…


Image for post
Image for post
Photo by Luke Chesser on Unsplash

One of the most awaited features, which was released by Databricks in Data & AI Summit 2020 SQL Analytics, and got me excited and I was all “Hell Yeah !!!”

Over the last two years, I have been evangelizing Databricks in the Architecture & Solution Design for the Unified Data Platforms.

For a Data Engineer, Databricks has proved to be a very scalable and effective platform with the freedom to choose from SQL, Scala, Python, R to write data engineering pipelines to extract and transform data and use Delta to store the data. …


Image for post
Image for post
Photo by Sai Kiran Anagani on Unsplash

For anyone in the area of Data & Analytics, busy in building a unified Data platform on the cloud within your organization, who is not aware of what Databricks is, it's time that you check it out before doomsday.

With almost 15 years in the Data Industry, having seen everything from the days of Traditional RDBM systems alongside low code or no code ETL tools to Hadoop and now Cloud-based Data platforms, Databricks has caught my attention and has excited me ever since I started using it 3 years back.

With more than 5000 customers, Databricks has its origins in…


A VPN works by routing your device’s internet connection through your chosen VPN’s private server rather than your internet service provider (ISP) so that when your data is transmitted to the internet, it comes from the VPN rather than your computer.

Image for post
Image for post
Photo by https://unsplash.com/@danny144

I was using NordVPN until recently and even though they have a strictly no-log policy but the only information we keep about our users is an e-mail address (used for connecting to VPN, marketing, and troubleshooting purposes) and the billing information (used for refunding procedures). But why should I even let them save that?

Here comes Outline VPN !!


Why is it important to have strong fundamentals for a Data Platform?

Having spent a couple of years now in the world of data, building end to end data management platforms right from the days, in the early 2000s, of Traditional databases like Oracle and SQL Server to the recent, days of Hadoop and real-time analytics using Databricks and Kafka, I have realized that, everywhere the problem statement is same and unfortunately the mistakes we make during the process of building the data platform are also the same.

So go ahead and read what I think are the top five mistakes we should avoid while building a Data Management Platform. …


Real-time data ingestion and making use of the data for creating reports for the business users have been on my mind for quite some time. As an Architect, I have always believed that the time to report between the moment data is captured in an organization and the time it is available for users to use it for taking decisions should be as less as possible. With the data in hand, it always becomes easy for business users to make informed decisions.

Working with multiple businesses in the last couple of years, there have been discussions and counter-arguments on

Image for post
Image for post

Do…


Image for post
Image for post
Photo by Steve Johnson on Unsplash

Azure Blob storage is an object storage solution for the cloud. Blob storage is optimized for storing massive amounts of data and you can use it for storing both structured and unstructured data like .csv, text, parquet, .json files and audio, video, and images.

While using Azure Storage account in an enterprise, you would always want to secure your storage account so that it is compliant as per your internal IT policies as well as regulatory compliance.

I won’t go into further details of the Storage Account here but would straight come to the point which I want to discuss…


Image for post
Image for post
Photo by Petter Lagson on Unsplash

Terraform is one of the most important tools in my tech stack to deal with the Cloud infrastructure. Gone are the days when I would use the mouse to click-click and deploy a network infrastructure or VM instances.

In this post, I would like to share my experience in deploying a very simple but Highly available network infrastructure. By High availability, I mean that we have a network with subnets in two availability zones in one region. So just in case one of the Availability Zones goes down, we will have our application running in the second availability zone.

This…

Mayank Srivastava

Data Architecture | Cloud Computing | Advanced Analytics

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store