Apache Pinot

Apache Pinot
Original authors
  • Kishore Gopalakrishna
  • Xiang Fu
DeveloperApache Pinot
Stable release
1.5.1 / 5 June 2026; 4 months ago (2026-06-05)
Written inJava
Operating systemCross-platform
Type
LicenseApache License 2.0
Websitepinot.apache.org
RepositoryPinot repository

Apache Pinot is an open-source, distributed, column-oriented, online analytical processing (OLAP), data store written in Java. Pinot is designed for low-latency analytical queries over data ingested from both streaming and batch sources. It is suited in contexts where fast analytics, such as aggregations, are needed at high-concurrency, and possibly, with real-time data ingestion.

Pinot was first created at LinkedIn in 2013 to support user-facing analytical reporting. Pinot supports real-time ingestion, SQL queries, distributed query processing, and multiple indexing techniques to improve query performance.

Active development on Apache Pinot now comes from contributors at StarTree (founded by Pinot's creators), Uber, Microsoft, Stripe, Walmart and Slack. Pinot is used by many more companies including Citi, Etsy, Hubspot, and Goldman Sachs

History

Pinot was started as an internal project at LinkedIn in 2013 to power a variety of user-facing and business-facing products. One of its early applications was LinkedIn's 'Who's used your profile' feature, which required analytics over recently generated data with low query latency.

The project was open-sourced in June 2015 under an Apache 2.0 license and was donated to the Apache Software Foundation by LinkedIn in June 2019. Pinot became an Apache top-level project in August 2021.

Apache Pinot 1.0 was released in August 2023. The release included Pinot's multi-stage query engine which extended it's SQL capabilities to include distributed operations such as joins, window-functions, and set operations. Development has continued with subsequent releases.

Architecture

Architecture of Apache Pinot
Architecture diagram of Apache Pinot

Pinot uses Apache Helix for cluster management. Helix is embedded as an agent within the different components and uses Apache ZooKeeper for coordination and maintaining the overall cluster state and health. All Pinot servers and brokers are managed by Helix. Helix is a generic cluster management framework to manage partitions and replicas in a distributed system.

Query management

Queries are received by brokers—which checks the request against the segment-to-server routing table—scattering the request between real-time and offline servers.

Cluster management

Pinot leverages Apache Helix for cluster management. Helix is a cluster management framework to manage replicated, partitioned resources in a distributed system. Helix uses Zookeeper to store cluster state and metadata.

Features

Pinot shares similar features with comparable OLAP datastores, such as Apache Druid. Like Druid, Pinot is a column-oriented database with various compression schemes such as Run Length and Fixed-Bit Length. Pinot supports pluggable indexing technologies - Sorted Index, Bitmap Index, Inverted Index, Star-Tree Index, and Range Index, which are what primarily differentiates Pinot from other OLAP datastores.

Pinot supports near real-time ingestion from streams such as Kafka, AWS Kinesis and batch ingestion from sources such as Hadoop, S3, Azure, GCS. Like most other OLAP datastores and data warehousing solutions, Pinot supports a SQL-like query language that supports selection, aggregation, filtering, group by, order by, distinct queries on data.

See also

References

  1. ↑ Cui, Tingting; Peng, Lijun; Pardoe, David; Liu, Kun; Agarwal, Deepak; Kumar, Deepak (14 August 2017). "Data-Driven Reserve Prices for Social Advertising Auctions at LinkedIn". Proceedings of the ADKDD'17. Association for Computing Machinery. pp. 1–7. doi:10.1145/3124749.3124759. ISBN 9781450351942. S2CID 12327343.
  2. ↑ Rosa, Marcello La (2021). ADVANCED INFORMATION SYSTEMS ENGINEERING: 33rd International Conference. Springer Nature. ISBN 978-3-030-79382-1.
  3. ↑ Chin, Francis Y. L.; Chen, C. L. Philip; Khan, Latifur; Lee, Kisung; Zhang, Liang-Jie (20 June 2018). Big Data – BigData 2018: 7th International Congress, Held as Part of the Services Conference Federation, SCF 2018, Seattle, WA, USA, June 25–30, 2018, Proceedings. Springer. p. 153. ISBN 978-3-319-94301-5.
  4. ↑ Im, Jean-François; Gopalakrishna, Kishore; Subramaniam, Subbu; Shrivastava, Mayank; Tumbde, Adwait; Jiang, Xiaotian; Dai, Jennifer; Lee, Seunghyun; Pawar, Neha; Li, Jialiang; Aringunram, Ravi (2018-05-27). "Pinot: Realtime OLAP for 530 Million Users". Proceedings of the 2018 International Conference on Management of Data. Sigmod '18. Association for Computing Machinery. pp. 583–594. doi:10.1145/3183713.3190661. ISBN 9781450347037. S2CID 44083085.
  5. 1 2 "The Apache Software Foundation Announces Apache® Pinot™ as a Top-Level Project". blogs.apache.org. 2 August 2021.
  6. ↑ Rogers, Ryan; Subramaniam, Subbu; Peng, Sean; Durfee, David; Lee, Seunghyun; Kancha, Santosh Kumar; Sahay, Shraddha; Ahammad, Parvez (16 November 2020). "LinkedIn's Audience Engagements API: A Privacy Preserving Data Analytics System at Scale". arXiv:2002.05839 [cs.CR].
  7. ↑ Javadi, Seyyed Ahmad; Gupta, Harsh; Manhas, Robin; Sahu, Shweta; Gandhi, Anshul (July 2018). "EASY: Efficient Segment Assignment Strategy for Reducing Tail Latencies in Pinot". 2018 IEEE 38th International Conference on Distributed Computing Systems (ICDCS). pp. 1432–1437. doi:10.1109/ICDCS.2018.00144. ISBN 978-1-5386-6871-9. S2CID 21659844.
  8. 1 2 3 Pawar, Neha. "Pinot Joins Apache Incubator" Archived 2019-04-02 at the Wayback Machine, LinkedIn Engineering, 01 April 2019
  9. 1 2 Gopalakrishna, Kishore. "Open Sourcing Pinot: Scaling the Wall of Real-Time Analytics". engineering.linkedin.com. LinkedIn. Archived from the original on 10 September 2015. Retrieved 3 September 2020.
  10. ↑ Yegulalp, Serdar (2015-06-11). "LinkedIn fills another SQL-on-Hadoop niche". InfoWorld.
  11. ↑ Fu, Yupeng; Soman, Chinmay (9 June 2021). "Real-time Data Infrastructure at Uber". Proceedings of the 2021 International Conference on Management of Data. Sigmod/Pods '21. Association for Computing Machinery. pp. 2503–2516. arXiv:2104.00087. doi:10.1145/3448016.3457552. ISBN 9781450383431. S2CID 232478317.
  12. ↑ "Apache Pinot Community Contributors". StarTree. 2021-08-09. Retrieved 2026-10-21.
  13. ↑ "Who Uses Pinot". Apache Pinot™. Retrieved 2026-10-06.
  14. ↑ Gopalakrishna, Kishore (Sep 19, 2023). "Grape to Glass: Apache Pinot™ 1.0 Uncorked".
  15. ↑ Ordonez, Carlos; Song, Il-Yeol; Anderst-Kotsis, Gabriele; Tjoa, A. Min; Khalil, Ismail (2 October 2019). Big Data Analytics and Knowledge Discovery: 21st International Conference, DaWaK 2019, Linz, Austria, August 26–29, 2019, Proceedings. Springer. p. 170. ISBN 978-3-030-27520-4.
  16. ↑ Uttamchandani, Sandeep (10 September 2020). The Self-Service Data Roadmap. O'Reilly Media, Inc. ISBN 978-1-4920-7520-2.