Data Eng Weekly (@DataEngWeekly) profile picture

Data Eng Weekly

@DataEngWeekly

Data Eng Weekly covers the week's top news in data engineering-related open source and cloud products. Curated by @joecrobak

dataengweekly.com
Joined August 2015
  • Tweets121
  • Following29
  • Followers991
Data Eng Weekly retweeted
Adam Kawa (@adam_kawa) profile picture
Adam Kawa@adam_kawa9 Sep 2019
To all my dear #BigData colleagues! Feel invited to submit an abstract to #BigDataTechWarsaw 2020, so that we can meet in Warsaw, talk about data and have a beer 🍺 This is the conference that I co-organise :) The CfP is open until Sep 30th. medium.com/getindata-blog/2%…

2³ Reasons To Speak at Big Data Tech Warsaw 2020 (February 27th, 2020)

Big Data Technology Warsaw Summit 2020 is fast approaching. This will be 6th edition of the conference that is jointly organised by…

medium.com
3
8
Data Eng Weekly retweeted
vicki (@vboykis) profile picture
vicki@vboykis15 Apr 2019
A couple of good links, both from @DataEngWeekly: 1. Review of idempotence complete with elevator analogy: lispcast.com/what-is-idempot… 2. How to learn to build distributed systems: brooker.co.za/blog/2019/04/0…

What is idempotence?

Idempotence means duplicates don't matter. It means you can safely retry an operation with no issues. The classic example is the elevator button: you press it twice and it does not call two elevato...

ericnormand.me
2
16
What changed in the Big data landscape from 2013 to 2019 blog.marouni.fr/bidata-trend… - interesting analysis from @abbassmarouni based on articles in @DataEngWeekly
Image from Robin Moffatt 🍻🏃🥓 (@rmoff)
Image from Robin Moffatt 🍻🏃🥓 (@rmoff)
1
11
1
18
Data Eng Weekly retweeted
Alberto Rodriguez (@ardlema) profile picture
Alberto Rodriguez@ardlema13 Dec 2018
My @apachekafka @kafkastreams challenges github repo has been announced in the last @DataEngWeekly issue: dataengweekly.com/Data-Eng-W… Thank you guys!
1
6
17
Replying to @dacort
Sorry about that! Did it end up in your spam folder by any chance?
1
List of new features in Amazon Redshift so far in 2018 - queries on nested data recently announced! blog.openbridge.com/whats-ne…
What’s New With Amazon Redshift?

What’s New With Amazon Redshift?

The features and capabilities of Amazon Redshift are constantly being updated by AWS. You can see the latest product, service, and feature…

blog.openbridge.com
1
Data Eng Weekly retweeted
Fabian Hueske (@fhueske) profile picture
Fabian Hueske@fhueske6 Sep 2018
Aaand another chapter is done! 🎉 The Early Release of "Stream Processing with @ApacheFlink" was updated with a new chapter about connectors and end-to-end consistency. Only two chapters ("Setup & Configuration", "Operations") are left. I should start looking for another hobby 🤔
Image from Fabian Hueske (@fhueske)
3
39
2
111
Data Eng Weekly 279 ▪︎ Benchmarking Hive on MR3 ▪︎ Event Sourcing ▪︎ Efficiently writing to a db ▪︎ Jepsen for Dgraph ▪︎ PulsarIO ▪︎ Scheduling of notebooks at Netflix ... and more! dataengweekly.com/Data-Eng-W…
1
5
Data Eng Weekly retweeted
Apache Airflow (@ApacheAirflow) profile picture
Apache Airflow@ApacheAirflow27 Aug 2018
Apache Airflow 1.10.0 is out ❤️🎉 !! Highlights: - New RBAC web interface in beta - First class kubernetes operator - Experimental kubernetes executor - Timezone support - Performance optimizations for large DAGs - Many GCP and S3 integration improvements - Tons of Bug Fixes
3
81
10
152
Data Eng Weekly retweeted
ApacheArrow (@ApacheArrow) profile picture
ApacheArrow@ApacheArrow7 Aug 2018
We've just released Apache Arrow 0.10.0, the biggest release yet with 4 months of work and nearly 500 issues closed. We've added 3 new languages to the project: Go, Ruby, and Rust. Read more arrow.apache.org/blog/2018/0…
3
47
82
Fortnite: 12 AWS regions, 5000 shards of Kinesis Streams, 4000 instances of EMR, and adding 2PB of data per month. datanami.com/2018/07/31/insi…
1
9
16
Catch up on the latest issue of Data Eng Weekly before the next one this Sunday! dataengweekly.com/Data-Eng-W…
1
2
Data Eng Weekly retweeted
Timo Walther (@twalthr) profile picture
Timo Walther@twalthr31 Jul 2018
The @ApacheFlink community reached 10,000 issues in JIRA. Thanks to everyone who participated! Let's file another 10,000 feature requests/bug reports and keep the community alive! issues.apache.org/jira/brows…
11
1
44
Big growth in flash storage: 55% year-over-year in Q1 2018. datanami.com/2018/07/26/the-…
1
Apache Accumulo 1.9.2 was released with fixes for a few serious bugs. accumulo.apache.org/release/…
5
1
Data Eng Weekly Issue #274 Lots of stream processing coverage this week—Apache Kafka, Wallaroo, Apache Samza, WSO2, and Amazon SQS + a couple of posts on Kubernetes, db monitoring + 2 new books + a proposed data ethics checklist. dataengweekly.com/Data-Eng-W…
1
8
ERA5 atmospheric data is now available on S3 as a public data set. Currently available from 2008 on-wards, all 9 petabytes dating back to 1950 will be released incrementally. medium.com/planet-os/era5-re…
ERA5 Reanalysis Data on S3

ERA5 Reanalysis Data on S3

During the past few weeks, in conjunction with the AWS Public Dataset Program, we have been working to bring reanalysis data to AWS…

medium.com
Speeding up ETL in PySpark by parallelizing db access. medium.com/@joaopedro.pinhei…
How to increase data extraction speed from database with PySpark

How to increase data extraction speed from database with PySpark

Scenario

medium.com
5
HDP 3.0 delivers new capabilities for the enterprise to enable agile application deployment, new #machinelearning /deep learning workloads, real-time database, & security and governance. Learn more about the enhancements, here: bit.ly/2JZms8t #BigData
6
12
Data Eng Weekly #273 is out. It was a tough one—so much great content to choose from. Coverage includes Scio, make at Propublica, Paypal's NameNode analytics, MySQL on Kubernetes, Kinesis+Lambda, data replication at Hotels.com, & much more. dataengweekly.com/Data-Eng-W…
3
Load more