Some Org mode workflow and graphs
@ Adrian Tritschler · Monday, Aug 14, 2017 · 4 minute read · Update at Aug 14, 2017 ·

Over the years far too many items catch my eye and have gone into the “one day/some day/maybe/perhaps”. Capturing them in Org mode was a major advance, at least then I had a record, but working through them is still a weak point. Some of them predate my use of Org mode, coming from bookmarks accumulated in del.icio.us, feedly, pinboard and manual links from prehistoric times.

Particularly problematic is an ever-growing (virtual) pile of PDF files, text files, EPUB books and bookmarked links, together with a fire-hose of RSS feeds. The latter is causing me some grief at the moment; the combination of org-feed pulling in feeds I’m interested, FOMO preventing me from being ruthless and culling them, and the website configuration of some sites causing org-feed to add a complete new copy of the last n sites every time it is run has conspired to cause my rss-org.org file to exceed 10MB – 105000 lines of text and 7690 items!

To try and get my head around what was piled up in the background I wrote a few snippets of shell, then decided to run them daily and graph the results. First up was the incoming bookmarks in notes.org, mostly captured from websites that have popped up during daily browsing. For some years I’d been using a capture template that pulled in items into the file as below:

* 2017
** 2017-08 August
*** 2017-08-02 Wednesday
**** [[http://sketchnotearmy.com/sa_books/][Books - A Showcase of Sketchnotes]]
    :PROPERTIES:
    :DateCreated: <2017-08-02 Wed 18:10>
    :KEYWORDS: sketchnote
    :ID:       015b9f93-fadf-44e7-ae47-039715724cc6
    :END:
     Source: [2017-08-02 Wed]

Then a grep "^\*"|wc -l give me a very quick and dirty count of items.

As soon as I had the current numbers I started kicking myself, wanting the historic values, so I reprocessed my git history:

for m in `seq 7`; do
  for d in `seq -w 28`; do
    (echo 2017-0${m}-${d};
      git show HEAD@{2017-0${m}-${d}}:notes.org |\
      grep "^\*\+ ${n}" |wc -l)\
      |paste -d "," - - | tee -a stats.csv
      done;
  done

Followed by a quick gnuplot template and I’ve got a graph of how high the pile of unread bookmarks is:

img

I put this into the daily cron job and let it run for a few days, then decided that I didn’t like having the year, month and day headings being counted so I’ve changed the capture template to add captured items as a “MAYBE” rather than a bare note, and a quick edit of the existing notes.org file to convert the existing bookmarks to MAYBE got me the sudden drop.

Some similar greps and and graphs and there’s a graph of the items waiting from my RSS feeds – clearly revealing that I’ve got a deal of housekeeping here.

img

As stated above, the RSS list has grown enormous and contains a mass of duplicated items. The drop from around 9000 to 7000 in mid-May was when I found that a previously behaving podcast feed had taken to republishing, daily, that the last 100 items were all “new” and fooled org-feed-update into adding each item again and again. More sorting and culling is required, when I get around to that there’ll be some similar drops in the item counts.

From there I decided to take a look at the breakdown of all tasks in all my org files; TODO, MAYBE, WAITING, STARTED… DONE. A slightly more complex bit of shell, grepping for "^\*\+ ${n}" where n is the keyword, piped out through wc(1) and munged into a CSV file. The numbers looked OK again I reprocessed my git history to find what it had looked like for the year to date.

To generate the graph took a few minutes of searching once I knew what I wanted and the correct terminology – stacked graph – to use. Courtesy of https://newspaint.wordpress.com/2013/09/11/creating-a-filled-stack-graph-in-gnuplot/ I now have a nice graph showing the stack of my TODO/MAYBE/STARTED..DONE tasks. et voila, we have:

img

The overall sawtooth shape is caused by me archiving off all completed work at the start of each month, the very large number of items in TODO and MAYBE is due to the presence of a number of extensive lists of items – eg a blog’s worth of articles turned into TODO items, or a list of someone’s “100 favourite books” – all present as low priority reading if I find sufficient time and interest.

In summary, I certainly found that visualising the changes in my captures, my RSS feeds and the state of my overall Org mode life through the year so far has given me some insight in how I’m using the tools.

Updates

  • [2017-08-17 Thu]: while removing vast numbers of duplicate entries in one of my RSS feeds I discovered that the overall graph hadn’t included the CANCELLED items so I added it to the statistics script and graph template, then reprocessed my git history to extract it as well. Two huge blocks of CANCELLED items show where I did some manual cleaning on the rss-org.org.
  • [2018-05-03 Thu]: I’ve collecting the same stats and generating the graphs into 2018, today I decided to to show them off.
Avatar of Adrian Tritschler

Adrian Tritschler's stuff

My website, an agglomerative mess, probably half-eaten by a grue
1960s 2016 2017 2018 2019 2020 2021 2022 2023 2024 250cc 2fa 30daysofbiking 3cr 3pbs 3rrr 403 404 4wd 9-11 a1000 a4000 aaac aabill aac abbotsford act additives adfa adsl adventure advertising afl agf agl airline airport ajft alley amazon ambulance amiga amiga1000 amiga4000 amys-ride angkor-wat anniversary antelope anzac apollo apple aps arboretum archicentre architecture armour aroundtuit art artwork ascension-records asus atbiad atomic attic au audax aus auspost aussiebirdcount australia australia-post australianbirds autoconfiguration autumn avatar avebury baaw baboon baby backup baklava ballarat balloon bandcamp bank barbecue bath bbq beach beer beeroclock beetle beijing benchmark benwerrin beryl bialetti bicycle bicycle-lane bigpond bigride bike-touring bikelane bikepath bikevic bingo bionicle bird birding birdingathome birds birdsseenin2023 birdsseenin2024 birthday bitcoin bite blackout blade blog blogging blogmax blogx blondie blossom blosxom bluestone bluetooth bmw bnsw boardgame boat boobs book bookmark books bouncy-castle bp breakdown breakfast brewery bridge bridge-road briefcase bright broadband brolga bromeliad brothel bsd buddy bugger-grips builder bulky bullant bund bungendore bunyip bureaucracy bus bus-lane bushwalking butterfly buzzword bv cactus cafe cake calculator callistemon cambodia camera cameraphone cameron camping cannondale canon cappuccino cappucino capuccino car carnegie casio cat caterpiller cats cbx cbx750 cbx750f cd cemetery censorship centro cereal chadstone challenge change cheese chef childhood chilli chimneyduck china chn chocolate christening christmas christmasbeetlecount church cicada cider cigarettebutts civet clayton clock clumsiness cm cms cockatoo cockroach coco coffee coffeemug coffeeneuring coffeeneuring2020 coffeeneuring2021 coffeeneuring2022 coffeeneuring2023 coffeeoutside coffeepot cold collision colnago comedy comet commonwealth-games commute commuting compiz computer concrete confusion connex coral corn couch court covid19 coworker crab crawl cremorne cricket crime critical-mass crossstitch crumpler css ctc ctw2850s curry custard customer-service cycle-path cycle-touring cycledindi cycledindi24 cycling cycliq cyclist cyclops dadjoke dalek dandenongs date death debian demo demo-scene demolition dentist derailment desk developer development dial-up dig digitisation digitise dinner disaster disgusting disk disqus distributed dizzy djerring-trail djerringtrail docbook dog dogs door dooring dorset dragon dragster dream driving drumming dtt duck dungeon durrell dvd dvico dyndns e-text e18 earworm easyshare echidna edge edge305 edge705 egret electronics elephant elevenses emacs email endomondo engagement england english engrish environment eofy eric-newby error erskine-river escooters esp estuary evening excom exif exiftool f3jr f3jv facebook fafnir fail2ban failure falcon family fancy fancy-dress fault fediverse feet fence festival fiat fiat-500 fiat500 fidel filter fire firefox firmware fish fishandchips fitbit fitzroy fixie flex flickr float flood floriade flower fly12 foaf fog food footy for-sale forums fotothing fountain fra fragile france freebsd friday friendship frog frogmouth frogs frost froty fruit fruittoast fungi gadabunud gadubanud gang-gang gansu garbage garden gardening garfish gariwerd garmin gas gbr geelong gemini genres geocache geocaching geography geotagged getoffmylawn gibberish gig gippsland giraffe girlfriend glass glen-waverley gmail gnome gnuplot goanna goat goatbeer golang goodreads goofey google gor gorse government gplus gps gpx graffiti grammar granfondo gravelbike gravelroad grddl greatsouthernrailtrail grenda groundeffect gru grub gsrt gsx1000z gtd guiness guinness guns gunzel gvbr gvrt gyroscope h1n1 habit hail hardrubbish hardware harvest hat hawkwind hdtv headwind health heatwave helicopter heraldsun hfbv hide hiking hippo history holiday honda honeymoon hope hospital hot hotel hrb hsfff huffy hug hughesdale hugo hume-highway humour huntingdale hyundai ibis ibm icon identity idiot ikea imagemagick imap incentivise incompetence indieweb inferno infographic injury ink inktapril inktob2020 inktober inktober2020 insect insects instagram intel interest internode intrepidtravel invertebrate invertebrates invitation iot iphone ipod iptc ipv6 iso8601 issue itch itsp itunes ixus ixus300 ixus700 jaywalker jekyll jersey jey jigsaw journal jrb june justice justjoey kangaroo kangaroos katana kawasaki kay-and-burton kde kellybrook kings-couriers kneejerk knitting koala kodak kookaburra kubernetes kudu ladder lake laneway laneways laptop lastfm lastpass laterpixel lawyer leather lego lemon lemons leslie-charteris lifelogging light lighthouse lilo limar limes linkedin linkrot linux lion litter lizard lj location lockdown logrotate lorne lroty lufthansa lunch lunchbreak lvm lxra lysterfield macchiato machine_tags machinereadymonday machinery magazine maglev magnolia magpie map mapping marksense markup marsupials matches mathjax mathml mattress maze mbtc mc6809 mc6809e md5 meerkat mekong melbourne melburn melburn-roobaix meme memolane memories memory meta metadata microformats microk8s microsoft millpond miniadventure minutae mist mlp mobile-phone modem mona monash monash-city-council monash-university monday mondegreen monitoring monument moon mooramong mortality motionbased motorbike motorcycle motorcycling motorist motorola movie movies mozilla mrtg msi mtb muarc mug mural murder murrumbeena muse muse-mode mushrooming music mycommute myflickryear2022 mysql names nanoadventure national-trust nbn neatstreets neighbour nest netbook netbsd netware network newspaper nigeria nightworks nocleanfeed nofilter noise nomnomnom noneshallpass nope norco norky-bike north-road nostalgia notebook notes novell nsw number1son nvidia nye nzl oakleigh obesity obituary obp obstruction oes2 offeeneuring oldcar ominous onenet open-tabs openbsd openid openindiana openphoto opensolaris opml oranges org-mode orgmode orgzly orly ostrich otter otways outage outpost owl oww oxford p100 palm-pilot panda paperwork parenting park parking parrot passbox pasta pbs pc pc3000 pc3121 pc3166 pc3232 pc3800 pentium pentium-4 pentium-iv peregrine perentie perl pesos pet petrol peugeot philosophy phone photo photograph photography photos php picasa picnic piggies pii piii pineapple piv pixelfed pizza plan9 plogging plu pobblebonk podargus podargus-strigoides poem police politics pollution portugal possum postcard postmarks postnuke pothole power ppp prawn prime probe problem proofreading prt pub pubs puck pumpkin pun punchcards puncture pyblosxom qantas qdos qfl qnx qotd quail quaxing r-class r707 radius raido rail-trail railtrail railway rain rainbow rainfall rainforest ramones random rant raven rc17 rdf rdup reading reading-books-rdf real-estate realestate recipe redevelopment renovation rent renting repair repairs reptile restaurant retrocomputing review rhino richmond rickroll ride2work ridetoworkday ring ringtail rip ritual river road-rage roadie roadrage roadsidefind robot rocket rockpool rockshox rodent rose roses roundabout rowing rrd rrdtool rrr rss rta rtabigride rtfm rtwd rubbish ruby ruins ruok russian-women samba samsonite samsung sasl sbs scam scanning scent schmap science scribble script sculpture sculptures seabird seaslug security self-reference selfie sensationalism sgml shakespeare shark shimano shoes shopping shopping-trolley sign signwriting silverbirch singing singlespeed sink site site-news site-stuff skateboard skink skink-link skip skyrail slashdot sleep sles sles10 slippery smarthome smidsy smog smoke smoker snake snakes sneakernet snow snow-train social-media software soloslog solstice soup souvenir spain spam spelling spider spiegeltent spotify spotty-bike spring springer ssh stagecoach station-trail steam-engine storage storm storm-trooper storytlr strava stupidity submarine suburbia summer sunos sunrise sunset supergirl supersunday supertuesday support surveillance survey suzuki svg switzerland syncthing t-shirt table tag tagging taikoz tailwind tandem tax taxi tdf teac teamrc17 technicomps technology teeth telemarketing telephone television telstra tent terrorism tgif the-saint the-slog theftbyfinding theme throw thunderstorm thursday ti ti99a til time toastie tomato touring tourism towel toys tradesman traffic trailgator train-spotter tram trap travel trek trek-t50 triffid tritschler trivia trovebox trustpilot tunnel turtle tv tvix twat-o-tron twitter typo ubuntu ugliness ui uk uluru unprecedented update upgrade uptime usb vandalism vcr vegemite veges vhs vic vicroads victoria video vietnam virus visitstanleytas vista visualisation vitriol vline vlocity vmware vpn vpnc walking wandrer warmshowers warranty wasp water water-meter waterfall wearegoingawol wearitpurple weather web webcam webfinger wedding weevil wellbeing wemo wetlands wfh whatcouldpossiblygowrong wideopenroad wifi wig wildebeeste wildlife wildoz win10 win2000 wind window windows windows7 windy wine winery winxp wired wisp witness wor wordplay work workflow worldbicycleday wpa www wyvern xemacs xml xp xrd xsl xslt yak yarrabug yellow ytbc yumcha zaf zebra zfs zoo zope zorse

© 1984 - 2024 Adrian Tritschler

Powered by Hugo with theme Dream.

About

author portrait

ajft looking stylish and black

…The Owner

There’s not much more I can add to who I am.

…The Site

Vanity site? Technology experiment? Learning tool? Blog? Journal? Diary? Photo album? I could tell you, but then I’d have to kill you…

I experiment. I play. I write and I take pictures. Some of the site is organised around topics, other parts are organized by date, then there’s always the cross-references between them.

Its all been here a fairly long time. Like the papers on my desk, or the books on the bedside table, the pile just grew… and it all grew without much plan or structure. I try not to break URLs, so historical oddities abound.

Long ago it started as a learning experiment with a few static HTML pages, then I added a bit of server-side includes and some very ugly PHP. A hand-built journal/blog on top of that PHP, then a few experiments in moving to various static publishing systems. I’ve never wanted a database-based blogging engine, so over the years I’ve tried PHP, nanoblogger, emacs-muse, silkpage and docbook before settling on Emacs Org mode for writing and jekyll for publishing. But the itch remained… I never really liked jekyll and the ruby underneath always seemed so much black magic. So now the latest incarnation is Org mode and hugo.

…The ISP

  • Hosted by @cos

…The Grue