Source (OpenNews)

Have a look at https://source.opennews.org - it's basically the watercooler for 'news nerds' (general term for newsroom developers/javascript journalists/dataviz people and so on). Not updated that frequently but often worth a read

Hi there, I'm the senior dev for NPR Visuals and the person who worked on White Lies (linked above, for which we did, in fact, use this guide). As a general rule, I don't comment here. But I did think this is worth responding to, because it speaks to the degree to which newsroom developers are still not visible even within the industry, much less outside. At this time, NPR's News Apps team is about five people, for a large national newsroom. That's actually pretty small! If you look at the kinds of teams that do this kind of work at places like the LA Times (which has both a graphics department and an award-winning Data Desk) or the NYT (last I checked, they had 40+ people on their graphics desk, plus an interactive team in editorial that works on projects like elections) or the Washington Post (big and still hiring). But many newsrooms (both print and broadcast) have at least small teams or single devs that work on big projects like this, even if they're not "flush with staff." That's because having an interactive team inside editorial is a force-multiplier for digital journalism--it lets us combine writing, video, audio, animation, and data in ways that makes the entire package better, so that hopefully people are more likely to pay for it. And if you have a team, you need a procedure for managing their resources and producing quality work, and that's all this document is. Sure, it's 38 pages long, but a few points: A) we don't follow this letter-by-letter on every…

This article compares multiple solutions and recommends docTR (Apache License 2.0): https://source.opennews.org/articles/our-search-best-ocr-tool-2023/

You do not want OCR for this. You want either ABBYY FineReader (around $99 for a license), or, if you prefer open source, Tabula: https://source.opennews.org/en-US/articles/introducing-tabula/ The main advantage of ABBYY is that if you need to do OCR, it is, in my opinion, the best consumer-level package. And it does a pretty good job of doing OCR and conversion to Excel. Here's a Github repo that demonstrates some results: https://github.com/dannguyen/abbyy-finereader-ocr-senate But to reemphasize, the above repo demonstrates ABBYY maintaining table structure with PDFs that are scanned images, which is considerably harder than the situation you're in. I've started a repo that eventually will compare text-to-table tools, which is what you want: https://github.com/dannguyen/pdftotablestable

NYT and Guardian have also put out great scrolling graphics over the last few days: http://www.nytimes.com/interactive/2015/10/27/world/greenland-is-melti… http://www.theguardian.com/us-news/ng-interactive/2015/oct/19/homan-sq… https://source.opennews.org/en-US/articles/how-we-made-homan-square-po…

I know of several startups in NYC using Go to great effect; bitly's nsq comes to mind: https://github.com/bitly/nsq The NY Times open sourced a very cool tool "streamtools" that was built in Go: https://source.opennews.org/en-US/articles/introducing-streamtools/ Your question also reminds me of an interesting talk I saw at RICON: https://speakerdeck.com/al3x/nobody-ever-got-fired-for-picking-java

We do actually have automated testing tools for that. https://bbc.github.io/wraith/ is one: see the write-up https://responsivenews.co.uk/post/56884056177/wraith or a contemporary interview https://source.opennews.org/articles/responsive-css-testing-made-simpl… . I get your overall point, though.

Looks good, but unfortunate name clash - I assume this is unrelated to the established Blacklight discovery platform used (ironically, considering The Markup's domain) in the Panama Papers exposure? https://projectblacklight.org/ https://source.opennews.org/articles/people-and-tech-behind-panama-pap…

Another tool for difficult-to-scrape sites is OCR. There are a few decent free/opensource options available: https://source.opennews.org/articles/so-many-ocr-options/

I found a relatively recent (Feb. 2019) comparison of OCR software and services [1]. It may be interesting to you, especially the "Texas Campaign Filing" comparison, which seems to have (nested) tables. [1] https://source.opennews.org/articles/so-many-ocr-options/

OK, we've changed to that from https://source.opennews.org/articles/how-propublica-illinois-uses-gnu-… Thanks!

Cool blog post about how NatGeo's 3D tour of the fossil was created using Three.js in the browser. https://source.opennews.org/articles/resurrecting-dragon/

Addendum: Methodology - http://graphics.wsj.com/hamilton-methodology/ OpenNews walkthrough of development/design process - https://source.opennews.org/en-US/articles/hamilton-algorithm/

I wonder if the people upvoting the article are actually reading it. It essentially just says "be careful" without any kind of practical advice for how to think like a statistician. Two much better articles: * https://source.opennews.org/en-US/learning/distrust-your-data/ * https://github.com/Quartz/bad-data-guide

It might not be exactly what you're looking for, but check out some of the listings here https://source.opennews.org/en-US/jobs/ or here http://www.newsnerdjobs.com/

The project is on GitHub at https://github.com/cantino/huginn You can also use Huginn to automate many other processes. It's basically an open source IFTTT. I also learned recently that it's being used at the New York Times: https://source.opennews.org/en-US/articles/open-source-bot-factory Here is another article about how to use Huginn to send you weather alerts: http://blog.andrewcantino.com/blog/2014/01/12/never-forget-your-umbrel…

Details about the redesign and technology overhaul: http://source.opennews.org/en-US/articles/behind-scenes-nyt-redesign/ i.e. "… using Github instead of SVN for version control, Vagrant environments, Puppet deployment, using requireJS so five different versions of jQuery don’t get loaded, proper build/test frameworks, command-line tools for generating sprites, the use of LESS with a huge set of mixins, a custom grid framework, etc."

Look for information you would want to see in your available media -- find a dataset and build it into a useful app to be filtered and queried in different ways. Map data points to physical locations. Correlate those things. At some point it all starts to look like a population density heat map, but sometimes you find good stuff in there. And people want to see it. They need more tangible access to all the real world data we have been collecting. Otherwise, what's the point? I worked for a year at a newspaper and built apps and stuff like that -- it was the most fulfilling work I've ever done and I wish the situation had been able to work out. If I ever get a chance to do real news apps again, I would jump on it. Enjoy your opportunity, Anders, and be prepared to show your work. Sometimes, the phrase "if you build it, they will come" is applicable. If you build something that shows you CAN build something, show it to the right people and you will attract attention. Some inspiration: http://blog.apps.chicagotribune.com/ http://blog.apps.npr.org/ https://source.opennews.org/en-US/ http://flowingdata.com/

The New York Times | New York, New York | Onsite | Full-time Come work in the newsroom of The New York Times and write code to shape the future of journalism. The Interactive News Team is looking for full-stack developers (ones with ops experience would be particularly welcome at this point) who care about the news and want to help make it better. We believe in work-life balance, shipping working products, and open source. Working in a newsroom isn't like working at a startup or a big company. Try it, you'll like it. Full job description: http://www.nytco.com/careers/newsroom/interactive-news-developer/ Apply by emailing [email protected] More about the NYT: https://news.ycombinator.com/item?id=9038615 And the journalism and code commnunity: https://source.opennews.org/en-US/organizations/new-york-times/ Our open source code: https://github.com/newsdev/ https://github.com/nytimes/ Other NYT tech jobs: http://www.nytco.com/careers/technology/