Couldn't we just store Instagram image CDN links instead of caching images?

[codelif]

What?

If we use instaloader without the args --no-metadata-json, instaloader will also store a json file with some metadata like display_url which contains the direct links to Instagram's CDN.

For example,

This link points to the post's image and can be accessed and streamed directly:

Why bother?

• Using this along with --no-pictures argument would save a lot of storage in the long term. (Repo size has already crossed 550 MBs)
• Instagram's CDN would be faster.
• Instaloader would also take less time because it doesn't have to cache the images.
• Metadata JSON also contains links for smaller-sized images and thumbnails which can be used to load the page faster in the future.

How much extra work would it be?

Not much. In terms of changes to the CD pipeline, just changing the args of instaloader would be enough.

For the code, for every post it would have to parse a JSON file to fetch the image URL.

I can create a proto today/tomorrow (if you want).

Potential Caveats

• Would make the project dependent on IG's CDN.
• If they switch CDNs, would need to update metadata cache as well.

[codeblech]

i see u fw this project
great effort brotha😎
the cdn thing...i was planning to do that (but fogor thanks for reminding)

If they switch CDNs, would need to update metadata cache as well.

do they do that regularly/frequently?

if you wanna go right ahead with a PR that'd be awesome 🔥
else I'd be on it anyways 🦾

[codeblech]

I'm opening an issue to track this

[codelif]

I am not sleeping tonight. Expect a PR when you wake up.

[codeblech]

😈

[codeblech]

gh notifications not working for me
hit me up on telegram/whatsapp if you need to ask anything urgent (though mention it on gh as well for the sake of open source)

[codelif]

i see u fw this project

I do.

do they do that regularly/frequently?

Not sure, probably not daily. I expect atleast 1 month (though I am pulling this out of my a**)

[codelif]

So..... The proto is done but.... it won't work.
The IG CDN has disabled Crossorigin requests. So it only allows requests from instagram.com

Sadly, I don't think we can fix this. All browsers support the CORS policies so this will be enforced on any W3C-compatible browser (which includes all of them)

demo.mp4

We could potentially make a "proxy server" which would stream these files for us, but that would induce too much latency.

Please suggest some solution.

[codeblech]

aight aight
can you think of a solution?
anyways, don't stress too much on it
we can find other ways of reducing storage usage, maybe?

[codeblech]

also, a bigger problem is running instaloader on the prod server. i tried it once, and all requests failed. I haven't yet figured out why.
if you like, you can work on this issue #10

on top of that, we need to setup a task on python anywhere that runs the instaloader command daily
i dont know if they allow cron jobs. that'd do the job too

[codelif]

can you think of a solution?

no, not currently. Some way or other, it leads to the server downloading the image.

we can find other ways of reducing storage usage, maybe?

Hopefully......
Compression maybe.

[codeblech]

I already use compression
I'm using cwebp to compress to webp

[codeblech]

i forgot to mention; that too, needs to be setup as an automated task