Northguard: Scalable Log Storage at Linkedin
There’s an announement for a new stream processing system tonight. Details below are copied from the meetup page and LinkedIn Event page:
Northguard is a log storage system developed at LinkedIn with a focus on scalability and operability. To achieve high scalability, Northguard shards its data and metadata, keeps minimal global state, and adopts a decentralized group membership protocol. Northguard’s operability leans on log striping to distribute load across the cluster evenly by design.
Speaker Bio: Onur is a Sr Staff Engineer at LinkedIn with an interest in distributed systems. He's the tech lead of Northguard, a log storage system with a focus on scalability and operability. Prior to Northguard, Onur was a committer to Apache Kafka, where he focused on Kafka's scalability. He redesigned the cluster's controller, made the controller use ZooKeeper's async APIs, and worked on the group coordinator and consumer group management protocol.
Along with the Northguard announcement, Xinfra (pronounced ˈzɪn.fɹə) is also getting a talk:
Northguard-Xinfra, LinkedIn’s virtualization system for Pub/Sub, offers a unified Pub/Sub experience for customers. Northguard-Xinfra is compatible to work with multiple log storage systems, including Kafka. Northguard-Xinfra provides a transparent switch between Pub/Sub systems, native federation and easier client management. Northguard-Xinfra achieves these goals by supporting a set of Pub/Sub agnostic APIs and having a dedicated metadata layer to provide a virtualization over heterogeneous Pub/Sub systems.
Speaker Bio: Wesley is a Sr Staff Engineer at LinkedIn. He joined LinkedIn Kafka team (now Streams IO team) in 2018. He worked on Kafka broker scaling and LinkedIn Kafka clients ecosystem previously. He currently focuses on the virtualized Pub/Sub System (a.k.a, Northguard-Xinfra).
Speaker Bio: Ke is a Staff Engineer at LinkedIn. He joined LinkedIn Kafka team (now Streams IO team) in 2018. He worked on Kafka clients streamline at LinkedIn and Kafka ecosystem previously. He currently focuses on the virtualized metadata/control layer in Northguard-Xinfra.
The talks are happening at:
If you’re in the Bay Area, you should come in person (maybe RSVP with one of the event links noted above). It’s going to be at:
LinkedIn Building 2
700 E Middlefield Rd.
Mountain View, CA 94043
Or, if you can’t make it in person, it’ll also be on Zoom.