About
Articles by Sandish Kumar
-
Processing the Streaming data Apache Spark Streaming
Processing the Streaming data Apache Spark Streaming
Processing the Streaming data has became very easy with Apache Spark Streaming Word Count on Streaming HDFS file: val…
1 Comment
Activity
20K followers
-
Sandish Kumar H N shared thishttps://lnkd.in/dwJqizz From phData, Inc. By Anthony Foerster #BigData #datapipeline #logging #solr #hadoop #cloudera
-
Sandish Kumar H N shared this#datascience #machinelearning #bigdata #phdataSandish Kumar H N shared thisHow many Machine Learning models does your organization have in production? In most organizations that's zero. This isn't a technology problem, it's a people and process problem. Learn more https://lnkd.in/eVCc6cd #machinelearning
-
Sandish Kumar H N shared thisKafka Topic Streams Table, must-read material https://lnkd.in/fnTkGyF
-
Sandish Kumar H N shared this#ApacheKuduSandish Kumar H N shared thisBrilliant Data has completed another production installation of Apache Kudu as a replacement for an expensive data warehousing solution (which is a trend we keep seeing in the marketplace). Talend, Streamsets, Knime and Tableau Server completed the overall solution package for reporting and advanced analytics.
-
Sandish Kumar H N shared thisSpark on Kubernetes! #spark #bigdata #kubernetes https://lnkd.in/e5StbBs
-
Sandish Kumar H N shared this@Jay Kreps 2013 post on Apache Kafka. Still interesting! #kafka #bigdataThe Log: What every software engineer should know about real-time data's unifying abstractionThe Log: What every software engineer should know about real-time data's unifying abstraction
-
Sandish Kumar H N shared this@phdata #bigdata #hadoop #analytics #machinelearning #opensource #jobs #bigdatajobs #sparkSandish Kumar H N shared this#Bigdata #Bigdatajobs #phdata Big Data is all we do. Are you inspired by innovation, hard work and a passion for big data? We're looking for proven performers with Hadoop Architecture, Administration & Optimization experience to create and deploy secure big data solutions for a diverse set of clients. Qualifications: 6+ years of Linux OS installation, configuration, administration and performance optimization as a Linux System or Java Middleware Engineer with emphasis on distributed computing. Experience integrating Linux OS with user authentication backend (LDAP/Active Directory) Hadoop experience including: Hadoop distribution (Cloudera preferred) including cluster installation and configuration Core Hadoop (HDFS, Hive, YARN) and on one or more ecosystem products/languages such as HBase, Spark, Impala, Search, Kudu, etc. Java application configuration and performance tuning Support for cloud-based big data clusters; AWS preferred Infrastructure automation; experience with Ansible and Git is a plus Interested Candidates please email your resume - bhagyashree@phdata.io
-
Sandish Kumar H N shared thisThorough Introduction to Apache Kafka https://lnkd.in/egF3Z2sThorough Introduction to Apache Kafka™ – Hacker NoonThorough Introduction to Apache Kafka™ – Hacker Noon
-
Sandish Kumar H N liked thisSandish Kumar H N liked thisI’m looking for an outstanding Engineering Manager (M1) for the Knowledge Distillation and Data Platform team in my org. This team is specifically focused on bridging the gap between massive Foundation Models and real-time production environments. We translate state-of-the-art foundation models into efficient, high-performance models that generate the lion's share of Meta's revenue. We are responsible for knowledge distillation and data augmentation research in collaboration with various teams, implementation and productization of those techniques at Meta’s scale, helping other teams utilize these techniques to productize their foundation models, and building and operating underlying services. Location: Sunnyvale (preferred), Bellevue, or Menlo Park. In This Role You Will: * Build, mentor, and grow a high-performing ML engineering team focused on knowledge distillation and foundation model productizations. * Partner with tech leads and managers from other teams, TPMs/PMs, and other stakeholders to define the long term strategic vision and roadmap. Bring in new knowledge distillation approaches and/or improve efficiency/scale to push the state-of-the-art further * Provide technical guidance and uphold engineering excellence through, emphasizing innovation, reliability, scalability, and ML practitioner experience. Foster a healthy, inclusive, and collaborative culture that focuses on impact and sustained growth. * Proactively identify and remove execution roadblocks to derisk expected outcomes * Represent the team internally and externally. You will thrive in this role if you have: * PhD in Computer Science/Engineering or a related field (or MS with equivalent relevant experience) * 10+ years of experience in ML engineering, including 5+ years of experience in engineering management * Very strong technical background and experience in Machine Learning (foundation models, knowledge distillation, data augmentation, online learning, LLMs, efficiency/optimization, etc.) * Built and led healthy team(s) of backend engineers, MLEs, and scientists * Passion for and experience in building and operating very large scale, production-critical, real-time machine learning applications/services. * Bias for action and don’t mind to getting deep into the trenches to help solve problems * Comfortably dealt with very large number of stakeholders, ambiguity, and rapid change * Have a strong track record of coaching engineers to achieve their peak potential * Exceptional communication skills * Great track record of strategic decision making as well as delivery under constraints. If you have passed the centralized hiring interview loop at Meta for M1, possess these strengths, and are interested in this area, please let me know. #AIML #Engineering #Leadership #AdsML
-
Sandish Kumar H N liked thisSandish Kumar H N liked this🎉 Thrilled to share that I’ve officially earned the AWS Certified Machine Learning – Associate! This milestone deepens my expertise in building, training, and deploying ML models on AWS using services like SageMaker, Glue, Lambda and more. Grateful for the learning journey and excited to apply these skills to create scalable, real-world AI solutions! #AWS #MachineLearning #Certification #CloudAI #DataScience
-
Sandish Kumar H N liked thisSandish Kumar H N liked thisDatabricks just announced the open sourcing of Real-time Mode (RTM) in Apache Spark Structured Streaming! Real-time Mode allows Structured Streaming queries to process data at super low latencies. Look at the benchmarks! The SPIP for this have already been accepted by the Spark Community: https://lnkd.in/gh6K8NEC I also gave two talks on Real-time Mode at DAIS if you would like to know more about this new feature. Real-Time Mode Technical Deep Dive: How We Built Sub-300 Millisecond Streaming Into Apache Spark™: https://lnkd.in/gYCJi2Tf Delivering Sub-Second Latency for Operational Workloads on Databricks: https://lnkd.in/gFdrSiir I think they are recorded? Feel free to contact me if you would like to know more or get started on using Real-time Mode! #dais #spark #streaming #databricks
-
Sandish Kumar H N liked thisSandish Kumar H N liked this🌎 Big month for team growth! We're excited to welcome 6 new talented professionals joining us from the US, Uruguay, and Argentina. Our Americas presence keeps getting stronger, and we couldn't be more thrilled to have this incredible group on board. Welcome to SunnyData! 🧡
-
Sandish Kumar H N liked thisSandish Kumar H N liked this📊 An SDE III at Amazon is *not* an L3 at Google. And a “Senior” at OpenAI isn’t the same as a “Senior” at Uber. Here’s how to make sure you don't get downleveled as a SWE 👇 💼 𝗦𝗗𝗘 𝗜𝗜𝗜 @ 𝗔𝗺𝗮𝘇𝗼𝗻 - Title: L6, Senior SDE - Median YOE: 10 - Median TC: $𝟯𝟴𝟲𝗞 💻 𝗟𝟯 @ 𝗚𝗼𝗼𝗴𝗹𝗲 - Title: SWE II - Median YOE: 2 - Median TC: $𝟭𝟵𝟯𝗞 Same number = same level, right? But one is a 𝗦𝗲𝗻𝗶𝗼𝗿 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿 with a decade of experience. The other? An 𝗲𝗻𝘁𝗿𝘆-𝗹𝗲𝘃𝗲𝗹 𝗦𝗪𝗘. --- Now let’s look at “Senior” engineers: 🧠 𝗟𝟰 @ 𝗢𝗽𝗲𝗻𝗔𝗜 (𝗦𝗲𝗻𝗶𝗼𝗿 𝗦𝗪𝗘) - Median YOE: 6 - Median TC: **$545K** 🚗 𝟱𝗮 @ 𝗨𝗯𝗲𝗿 (𝗦𝗲𝗻𝗶𝗼𝗿 𝗦𝗪𝗘) - Median YOE: 8 - Median TC: **$464K** Same title—“Senior”—but different levels, and different notations. And don’t even get us started on how Uber uses Senior → Staff → Senior Staff leveling unlike OpenAI which uses straight numbers… --- In comes Levels.fyi—a tool to bring transparency to a world where a title like "Level 3" or "Senior" tells you almost nothing. What actually matters: ✅ What level are you? ✅ How many years of experience do peers at that level have? ✅ What’s the compensation? Because titles don’t scale. Compensation does. View leveling and salary information here: https://lnkd.in/gu9-Sx_U 👇 Ever been confused by SWE level names? Drop a comment!
-
Sandish Kumar H N liked thisSandish Kumar H N liked thisTotally wowed by Gemini 2.5 Pro this evening. I dumped a fairly complicated few thousand lines of c++ into it and asked it to explain everything to me, and it was both more complete than our internal documentation and had better grasp of the details than I did. I then had a very cogent conversation about potential improvements to the data structures, and felt like I was talking to a peer.
-
Sandish Kumar H N liked thisSandish Kumar H N liked thisIn my career I've looked at some pretty tricky bugs... among the hardest are (a) non-deterministic bugs that reproduce rarely, and (b) miscompilation bugs, where the compiler generates incorrect code despite being fed correct source. Today was a first -- tracked down a non-deterministic miscompilation, in which LLVM would generate incorrect optimized code about 1% of the time for a certain code shape with certain set of optimizations applied! Unfortunately, even safe languages aren't immune to such a mess. I hope by the time my kids are debugging systems we'll have figured out solutions to these things...
-
Sandish Kumar H N liked thisSandish Kumar H N liked thisWelcome to all our February new hires! We’re excited to have you join the phData team as we start another great year. We look forward to supporting your growth and celebrating your successes! 🎉 #phData #NewHires #WelcomeToTheTeam
Experience & Education
Licenses & Certifications
Publications
-
Visualizing NetFlow Data with Apache Kudu, Apache Impala (incubating), StreamSets Data Collector, and D3.js
phdata
See publicationVisualizing NetFlow Data with Apache Kudu, Apache Impala (incubating), StreamSets Data Collector, and D3.js
-
Visualizing NetFlow Data with Apache Kudu, Apache Impala (incubating), StreamSets Data Collector, and D3.js
StreamSets
See publicationVisualizing NetFlow Data with Apache Kudu, Apache Impala (incubating), StreamSets Data Collector, and D3.js
Projects
-
Yardstick Spark
Technologies Used: Spark , Spark SQL CoreData Frame's, Spark Streaming, Scala, HDFS, Apache Ignite, Yardstick Tool, D3js, Tableau, AWS
Data Sets: 10 million twitter data records and 1 billion auto generated records.
Description:
Yardstick Apache Spark is a set of Apache Spark benchmarks written on top of Yardstick
framework.
I have successfully written Spark CoreRDD application to read auto generated 1
…Technologies Used: Spark , Spark SQL CoreData Frame's, Spark Streaming, Scala, HDFS, Apache Ignite, Yardstick Tool, D3js, Tableau, AWS
Data Sets: 10 million twitter data records and 1 billion auto generated records.
Description:
Yardstick Apache Spark is a set of Apache Spark benchmarks written on top of Yardstick
framework.
I have successfully written Spark CoreRDD application to read auto generated 1
billion records and compare with IgniteRDD in Yardstick framework to measure
performance of Apache Ignite RDD and Apache Spark RDD.
I have successfully written Spark DataFrame application to read from HDFS and analyze
10 million twitter records using Yardstick framework to measure performance of Apache
Ignite SQL and Apache Spark DataFrame.
I have successfully written Spark Streaming application to read streaming twitter data and
analyze twitter records in real time using Yardstick framework to measure performance of
Apache Ignite Streaming and Apache Spark Streaming.
Implemented test cases for Spark and Ignite functions using scala as language.
Hands-on experience in setting up 10 node Spark cluster on Amazon Web Service’s
using Spark EC2 script.
Implemented and D3.js Tableau charts to show performance difference between Apache
Ignite and Apache Spark.
Other creatorsSee project -
Comparative Analysis of Big Data Analytical Tools –(Hive,Hive on Tez,Impala,SparkQL,Apache Drill, BigQuery, PrestoDB running on the Google Cloud and AWS)
Technologies Used: Hive,Hive on Tez,Impala,SparkSQL, Apache Drill, BigQuery, PrestoDB, Hadoop, Cloudera
CDH, Hortonworks HDP, Google Cloud Platform, Amazon web service
Data set used: Twitter streaming data
a. Installation of Hive,Hive on Tez,Impala, SparkQL, Apache Drill, BigQuery, PrestoDB, Hadoop,
Cloudera CDH, Hortonworks HDP
b. Schema design for data sets on all Hive,Hive on Tez,Impala, SparkQL, Apache Drill, BigQuery,
PrestoDB
c. Query design for given data set
d…Technologies Used: Hive,Hive on Tez,Impala,SparkSQL, Apache Drill, BigQuery, PrestoDB, Hadoop, Cloudera
CDH, Hortonworks HDP, Google Cloud Platform, Amazon web service
Data set used: Twitter streaming data
a. Installation of Hive,Hive on Tez,Impala, SparkQL, Apache Drill, BigQuery, PrestoDB, Hadoop,
Cloudera CDH, Hortonworks HDP
b. Schema design for data sets on all Hive,Hive on Tez,Impala, SparkQL, Apache Drill, BigQuery,
PrestoDB
c. Query design for given data set
d. Debugging on Hive,Hive on Tez,Impala, SparkQL, Apache Drill, BigQuery, PrestoDB, Hadoop,
Cloudera CDH, Hortonworks HDP
e. Time Comparison of each Hive,Hive on Tez,Impala, SparkQL, Apache Drill, BigQuery, PrestoDB
f. Time comparison between different cloud platforms
g. Times metrics web based visualization design on google charts
Other creatorsSee project -
Truck Events Analysis
Technologies Used: Hadoop, HDFS, Hive, HBase, Kafka, Storm, RabbitMQ WebStormp, Google Maps
Data Sets Used: - New York City Truck Routes from NYC DOT. -Truck Events Data generated using a custom
simulator. - Weather Data, collected using APIs from Forcast.io. -Traffic Data, collected using APIs from
MapQuest.
a. Written a simulator to send/emit events based on NYC DOT data file.
b. Written Kafka Producer to accept/send events to Kafka Producer which is on Storm Spout
c…Technologies Used: Hadoop, HDFS, Hive, HBase, Kafka, Storm, RabbitMQ WebStormp, Google Maps
Data Sets Used: - New York City Truck Routes from NYC DOT. -Truck Events Data generated using a custom
simulator. - Weather Data, collected using APIs from Forcast.io. -Traffic Data, collected using APIs from
MapQuest.
a. Written a simulator to send/emit events based on NYC DOT data file.
b. Written Kafka Producer to accept/send events to Kafka Producer which is on Storm Spout
c. Written Storm topology to accept events from Kafka Producer and Process Events
d. Written Storm Bolt to Emit data into Hbase, HDFS, RabbitMQ WebStomp
e. Hive Queries to Map Truck Events Data , Weather Data, Traffic Data
Other creators -
Cimbal/MobApp Pay
Cimbal is a mobile promotion and payment network designed to increase business sales and deliver targeted deals to
consumers.
Technologies used: Hadoop MapReduce, Hbase , Spring Data Rest Web Service, CDH
Data Set : Users Payment Data
a. Written MapReduce programs to validate the data
b. Written more than 50 Spring Data Hbase rest API's in Java
c. Schema design on Hbase and cleaning data
d. Written Hive queries for analytic's on users data.
Other creatorsSee project -
Social Media Sentiment Analysis Update on each 4 hours (Positive Negative Review's)
Social Media Sentiment Analysis project gives positive negative reviews on our favorite words search, it is based on Twitter tweets, Facebook status's etc... it has been implemented using Social media API's with BigData Technologies like Flume to get social media stream data regularly into HDFS and Cassandra, MongoDB to store sentiment analyzed data. and in this project i am using apache oozie to develop work flow and it will be ruing on each 4 hours. this is project is done with my self…
Social Media Sentiment Analysis project gives positive negative reviews on our favorite words search, it is based on Twitter tweets, Facebook status's etc... it has been implemented using Social media API's with BigData Technologies like Flume to get social media stream data regularly into HDFS and Cassandra, MongoDB to store sentiment analyzed data. and in this project i am using apache oozie to develop work flow and it will be ruing on each 4 hours. this is project is done with my self interest.
-
Whole Genome Sequencing - Quality Check (QC)
See project-Entire program was developed in Hadoop MapReduce praradigm
-Performed custom Quality Check on genomic data to retain good quality reads
-Implemented novel features in the software:
-capability to handle sequencing-machine errors
-automatic detection of base-line PHRED score
-being platform agnostic and able to handle various file-formats i.e. from Illumina, 454 Roche, Complete Genomics, ABI Solid input format data -
Whole Genome Sequencing - Sequence Alignment
See project- Developed a Hadoop MapReduce program to perform sequence alignment on NGS data.
- The MapReduce program implements algorithms such as Borrows-Wheeler Transform (BWT), Ferragina-Manzini Index (FMI), Smith-Waterman dynamic programming algorithm using Hadoop distributed cache. -
Seismic Data Server & Repository (SDSR™)
Our Seismic Data Server & Repository solves the problem of delivering, on demand, precisely cropped SEG-Y files for instant loading at geophysical interpretation workstations anywhere in the network. Based on Hadoop file storage, Hbase™ and MapReduce technology, the Seismic Data Server brings fault-tolerant petabyte-scale store capability to the industry. Seismic Data Server supports post-stack traces now with pre-stack support to be released shortly.
Other creatorsSee project -
DDSR (Drilling Data Search and Repository):
This project aims to provide analytics for Oil and Gas exploration data. This DDSR repository builded by using HBase, Hadoop and its sub projects. We are collecting thousands of wells data from across the globe. This data is stored in Hbase and Hive by using Hadoop MapReduce jobs. On top of this data we are building analytics for search and advanced search.
Other creatorsSee project -
E Commerce (Obesessory.com)
-
See projectTech: Hsearch(Hbase+lucene), Hbase, Cassandra, Hive, Spark, Hadoop, MapReduce,Amazon Web
Service, Linode, CDN, Scala, Java
Data: Affilats feeds Rakuten, CJ, Affiliate window, WebGains.
Pre-Processing:
Crawling of 100+ sites Data using Nutch
Fashion based ontology maintenance
Using Spark, Scala:
to Enriched given data using Fashion Ontology
to Validation/Normalizing the data
Designed schema, Modeling the data and Written program to store all validated data in…Tech: Hsearch(Hbase+lucene), Hbase, Cassandra, Hive, Spark, Hadoop, MapReduce,Amazon Web
Service, Linode, CDN, Scala, Java
Data: Affilats feeds Rakuten, CJ, Affiliate window, WebGains.
Pre-Processing:
Crawling of 100+ sites Data using Nutch
Fashion based ontology maintenance
Using Spark, Scala:
to Enriched given data using Fashion Ontology
to Validation/Normalizing the data
Designed schema, Modeling the data and Written program to store all validated data in Cassandra
Spring Data Cassandra Programs for Validation/Normalizing/Enriching and REST API to Develop UI Based manual QA Validation.
Used SPARQL, Scala to running QA based SQL queries.
Indexing:
MR Programs on Hbase:
To Standardize the Input Merchants data
To Upload images to RackSpace CDN
To Index the given Data sets into Hsearch
To MR Programs on Hbase to extract the color information from Images including density
To MR Programs on Hbase to Persist the Data on Hbase tables
above MR jobs will run based on timing and bucketing
Color-Obsessed:
Using Image color and density data
User will be allowed to select 1,2.. colors with different densities and result will be a list of
products where each product image contains all give colors with exact density
Written Hbase Spring rest web service for Color Obsessed Search API
Post-Processing:
Setting up the Spark Streaming and Kafka Cluster
Developed a Spark Streaming_Kafka App to Process Hadoop Jobs Logs
Kafka Producer to send all slaves logs to Spark Streaming App
Spark Streaming App to Process the Logs with given rules and produce the Bad Images,Bad
records, Missed Records etc
Spark Streaming App collect user actions data from front end
Kafka Producer based Rest API to collect user events and send to Spark Streaming App
Hive Queries to Generate Stock Alerts, Price Alerts, Popular Products Alerts,
New Arrivals for each user based on given likes, favorite,shares counts information
Working on SparkMLlib for Recommendations, Coupons Recommendations, Rules
Engine
Languages
-
English
Professional working proficiency
-
Telugu
Native or bilingual proficiency
-
Kannada
Native or bilingual proficiency
-
Tamil
Limited working proficiency
Recommendations received
-
LinkedIn User
4 people have recommended Sandish Kumar
Join now to viewOther similar profiles
Explore top content on LinkedIn
Find curated posts and insights for relevant topics all in one place.
View top content