get-posts-stateless All Flags Set to False
by kitty4d
so i had asked in the DeSo Discord about if there was some kind of hardcoded limit to the amount of records that would get returned by get-posts-stateless. since it’s really for the feeds, i thought maybe it did have a limit, since who would ever scroll the feeds to the beginning of the blockchain? and it seemed like my code to retrieve the records was ending without getting all that many records. anyway- i’ve got the Discord chat here. and after that, i’ll include my updated code too, cause i’m pretty sure i had a problem with the code before.
DeSo Foundation Tech - DEVELOPERS > dev-support | Updated Kitty4D JavaScript
so, because i don’t want it to be hard to find any information, these seem to be the key takeaways. none of these things are documented anywhere so, hopefully this help someone else not have to ask these questions!
- get-posts-stateless, with none of the flags set to true, will return ALL posts if you keep paginating until you reach the end. in reverse chron order. this was my assumption, but it’s not stated anywhere.
- following feed is only computed for the past 2 days - i don’t believe this is documented.
- the feed by DESO goes only an hour back - this is documented.
- NumToFetch is not necessarily going to match the # of top-level post records you get back. i believe this is primarily due to reposts counting as additional posts in the response. it’s possible it’s also due to hidden posts, though i feel like that SHOULD be dealt with in the backend. but i don’t know for sure. either way don’t be shocked to get 35 records when you request 50.
- the deprecated fields like RecloutedPostEntryResponse and numeric counts for Reclouted/QuoteReclouted will contain the same data as the newer-named fields. the docs don’t mention the old fields in the response type documentation, it threw me off for a second.
- addGlobalFeedBool is documented as adding the isGlobal bool in the response, but it also controls the inHotFeed bool.
doing the call with none of the “feed flags” (GetPostsForFollowFeed, GetPostsForGlobalWhitelist, GetPostsByDESO) is the “get all posts ever” call. using any of the flags makes it into a much different call, because you cannot go forever with any of the 3 feeds.
when doing the “get all posts ever”, the default OrderBy is “newest” (reverse chron) - and i believe (this i have not confirmed 100% but it makes sense to me, while also not making total sense) - the pagination of posts only works in this order. and results ONLY come out in this order technically. if i fetch 50 posts at once, i’m getting those 50 posts in the reverse chron order based on where i set my starting point at - and any change of OrderBy is only done AFTER the fact. so, i can get the most recent 50 posts, and i can then order those 50 posts in “oldest” order, but you can’t simply do orderBy: oldest and fetch 50 posts with no postHashHex and get the absolute 50 first posts ever made on the blockchain. blame it on the linked list? i’ve tested this multiple times and i feel that this is correct but i have never done something like: grab 50 posts, set order by to oldest, then try to retrieve the next page of data using what is now the last element’s posthashhex in those results. i don’t think it would work though, at least not in the expected way (if your expectation is that the next page is going to be the next 50 NEWEST posts after that post, since ordering by oldest would cause that - i think it will be the 50 next oldest posts).
i also have not attempted to use OrderBy with any of the feeds. i don’t see why it wouldn’t work, but probably in the same fashion. will confirm this stuff eventually!
DeSo Foundation Tech - DEVELOPERS > dev-support
is there a fixed max number of posts that can get returned by get-posts-stateless? or maybe a max amount of days back it can go? i don't mean per request. i got about 1135 posts and then the next request brought back an empty array.
kitty4D
is there a fixed max number of posts that can get returned by get-posts-stateless? or maybe a max amount of days back it can go? i don't mean per request. i got about 1135 posts and then the next request brought back an empty array.
you can paginate by PostHashHex or StartTstampSecs I believe, there is also NumToFetch param
Kuririn
you can paginate by PostHashHex or StartTstampSecs I believe, there is also NumToFetch param
yeah i am paginating on it, but it doesn't seem to go back very far. i mean, i got like 10000 posts but that's not every post that's ever been made. so i was wondering if there is a limit- i couldn't see one hardcoded in the backend endpoint code but i didn't go super deep in it either
kitty4D
yeah i am paginating on it, but it doesn't seem to go back very far. i mean, i got like 10000 posts but that's not every post that's ever been made. so i was wondering if there is a limit- i couldn't see one hardcoded in the backend endpoint code but i didn't go super deep in it either
@kitty4D - We only compute the past two days for the following feed and we don't allow getting posts by DESO for more than a hour back.
lazynina
@kitty4D - We only compute the past two days for the following feed and we don't allow getting posts by DESO for more than a hour back.
thanks- i'm not setting either of those flags to true, nor globalWhiteList, so what exactly IS it returning to me? my assumption was that it would just be the most recently-added posts
kitty4D
thanks- i'm not setting either of those flags to true, nor globalWhiteList, so what exactly IS it returning to me? my assumption was that it would just be the most recently-added posts
Okay so you'll be getting ALL posts with no filtering, except removing users blocked by ReaderPublicKeyBase58Check
lazynina
Okay so you'll be getting ALL posts with no filtering, except removing users blocked by ReaderPublicKeyBase58Check
okay - i'm not giving it a reader. but is the idea that it really would eventually give me every post? maybe the backend doesn't like me requesting so much data (i am caching it at least so i make the request once and generate static html and don't make the same request again) - i got about 11,000 posts using it. i probably should have given it an orderBy, but i didn't, lol
kitty4D
okay - i'm not giving it a reader. but is the idea that it really would eventually give me every post? maybe the backend doesn't like me requesting so much data (i am caching it at least so i make the request once and generate static html and don't make the same request again) - i got about 11,000 posts using it. i probably should have given it an orderBy, but i didn't, lol
Not ordering it should return them in chronological order. You should be able to go back in time forever with this
lazynina
Not ordering it should return them in chronological order. You should be able to go back in time forever with this
thanks! i am actually glad it didn't go forever, but i'll try starting off from my last postHashHex and put in a limit on how many posts it would get before giving the API a break.
i'm pretty sure my last request did just return { 'PostsFound': [] } though - not any kind of error that i could see. but i don't really want to be hitting an API long enough to get more than, like, 1000 posts at a time anyway
kitty4D
thanks! i am actually glad it didn't go forever, but i'll try starting off from my last postHashHex and put in a limit on how many posts it would get before giving the API a break.
i'm pretty sure my last request did just return { 'PostsFound': [] } though - not any kind of error that i could see. but i don't really want to be hitting an API long enough to get more than, like, 1000 posts at a time anyway
that's strange! Send me the post hash hex that resulted in the empty result if you can find it please
1
Hi @lazynina would be nice to have NumToFetch while using /api/v0/get-nfts-for-user Or there's a way to do this already?
Bobnificent
Hi @lazynina would be nice to have NumToFetch while using /api/v0/get-nfts-for-user Or there's a way to do this already?
unfortunately the issue with this endpoint is that we're grouping serial numbers of the same NFT and there's no logical way to order the NFTs. If you had a scheme to support NumToFetch, I would welcome a PR
@lazynina so i think my issue before was with my code, but now as i hit the get-posts-stateless endpoint, i get very erratic results. i'm requesting 50 at a
time, but the actual number of results i get varies from maybe as low as 24 (but more often in the low-to-mid-30s) up to 50. very inconsistent. i've only seen it return 50 maybe 10% of the time. attaching some images with postHashHex's and
the recordcount that came, when numToFetch is 50. i'll type them out and paste them in here in a minute.
b0dc87386abd2103c91e7623123595d5b08fec5db47f20eb73f8f89b577de668.. ohhh
could it be reposts
feel like i might have read that somewhere, that Reposts count towards the total count of posts that get returned
i guess it's because in the backend, it's counting the total number of postEntryResponses it has to retrieve.
ok lol new question, this is for anyone
oh duh nvm let me delete all this i didn't post anything
i guess both Reposted and Reclouted get populated always and i got mixed up in my JSON. i never noticed that before though, i must be blind
- looks like addGlobalFeedBool also controls if you get back inHotFeed also. just making note of it here.
if I remember reading the comments in the code, there is something about the count not being exact because they fetch things, then remove blocked, hidden or whatever other rules content.
yeah, i'm not passing a readerPk so no blocked. i'll look at the code again for hidden, but that would be nuts that there would be between 10-20 hidden posts in every batch of 50. the json looked to mesh with the reposts being counted as unique posts
OlivierOzoux
if I remember reading the comments in the code, there is something about the count not being exact because they fetch things, then remove blocked, hidden or whatever other rules content.
it looks like from "true /skipHidden/," that a param is getting passed that would be removing the hidden posts before it would be making a count, but i'll dig a little more (https://github.com/deso-protocol/backend/blob/709cbfbc62cf3a0e6d56c393e555fc277c93fb76/routes/post.go#L441)
but i dunno i don't know this go language well enough to tell once i get into the loop in GetPostsByTime.
but what is lib.DBGetPaginatedPostsOrderedByTime
oh ok that might be in another castle
oh ok badgerDb that sounds super cute, that's interesting i guess i just assumed the blockchain was just like binary data floating around on the disk lol, i hadn't really thought about it that hard
1
lazynina
unfortunately the issue with this endpoint is that we're grouping serial numbers of the same NFT and there's no logical way to order the NFTs. If you had a scheme to support NumToFetch, I would welcome a PR
Ok. Thanks for the reply.
okay so here’s my current code that uses get-posts-stateless. i realized that the problem i was having before might have been that when i abstracted out my call for fetch to make a fetchCache function, i didn’t make that function async. i’m just learning the async stuff so, hopefully i’m doing it right now. i think i can download all records now if i let it, but i have a limit hardcoded in now.
also to note, i have a blacklistedUsers array, i’m not showing it since the public keys are for users whose data i don’t want to share.
so i had left this function out before actually. before i didn’t have it async with awaits. i dunno if i really need these on everything but like i said, seems to work so lol.
async function fetchCache(url, params) {
/* pseudoGetParams will be appended to the URL in order to maintain a cache per unique API call
otherwise, a call to get the number of followers for a user and a call to get the number of followings for a user will
be cached as a shared object, which is not the expected behavior.
i had expected the caching to take into account POST parameters the same way it does GET parameters, but
since it doesn't, this seems to be a working solution.
*/
const pseudoGetParams = params.body ? encodeURIComponent(params.body) : '';
let res = await Cache(url + '?' + pseudoGetParams, {
duration: "1d",
type: "json",
fetchOptions: params,
});
return await res;
}i’ll show a bit more of the file now, with the class definition and stuff.
class DesoLite {
constructor({ apiurl = default_apiurl, pkey = '', userName = '' }) {
this.apiurl = apiurl;
this.pkey = pkey;
this.userName = userName;
}
async getPosts({ pk = this.pkey, postHashHex = null, orderBy = '', postContent = '', numToFetch = null, fetchSubComments = false, getPostsForFollowFeed = false, getPostsForGlobalWhitelist = false, getPostsByDESO = false, mediaRequired = false, postsByDESOMinutesLookback = null, addGlobalFeedBool = false }) {
let url = this.apiurl + 'get-posts-stateless';
try {
let res = await fetchCache(url, {
method: 'POST',
headers: {
'Content-Type': 'application/json'
},
body: JSON.stringify({
ReaderPublicKeyBase58Check: pk,
PostHashHex: postHashHex,
OrderBy: orderBy,
StartTstampSecs: null,
PostContent: postContent,
NumToFetch: numToFetch,
FetchSubcomments: fetchSubComments,
GetPostsForFollowFeed: getPostsForFollowFeed,
GetPostsForGlobalWhitelist: getPostsForGlobalWhitelist,
GetPostsByDESO: getPostsByDESO,
MediaRequired: mediaRequired,
PostsByDESOMinutesLookback: postsByDESOMinutesLookback,
AddGlobalFeedBool: addGlobalFeedBool
})
});
return await res;
} catch (error) {
console.log(error);
}
return null;
}
async getFrom(method, params = {}) {
let res;
switch (method) {
case 'get-posts':
/* for Posts, pass:
pk: reader's public key,
postHashHex: for paging, send last received post hash,
orderBy: newest|oldest|last_comment,
postContent: only return posts whose (body only?) matches the text,
numToFetch: non-zero,
fetchSubComments: return 1 level deep of comments? haven't tested yet,
getPostsForFollowFeed: follow feed for reader's pk,
getPostsForGlobalWhitelist: global posts, i guess this is node-agnostic?,
getPostsByDESO: kinda basically the hot feed i guess,
mediaRequired: false, or true but would be nice to specify image and/or video
postsByDESOMinutesLookback: must be <= 60, and only when doing getPostsByDESO - dunno why it doesn't work for all.. ,
addGlobalFeedBool: posts will contain a bool (but i kinda thought they already had an IsGlobal..?)
*/
try {
const batch = await this.getPosts({ pk: params.readerPk ? params.readerPk : undefined, postHashHex: params.postHashHex ? params.postHashHex : undefined, orderBy: params.orderBy ? params.orderBy : undefined, postContent: params.postContent ? params.postContent : undefined, numToFetch: params.numToFetch ? params.numToFetch : 10, mediaRequired: params.mediaRequired ? params.mediaRequired : false });
res = batch.PostsFound.filter(item => !blacklistedUsers.includes(item.PosterPublicKeyBase58Check));
} catch (error) {
res = [];
}
break;
case 'get-posts-all':
/* for Posts, pass:
pk: reader's public key,
postHashHex: for paging, send last received post hash,
orderBy: newest|oldest|last_comment,
postContent: only return posts whose (body only?) matches the text,
numToFetch: non-zero,
fetchSubComments: return 1 level deep of comments? haven't tested yet,
getPostsForFollowFeed: follow feed for reader's pk,
getPostsForGlobalWhitelist: global posts, i guess this is node-agnostic?,
getPostsByDESO: kinda basically the hot feed i guess,
mediaRequired: false, or true but would be nice to specify image and/or video
postsByDESOMinutesLookback: must be <= 60, and only when doing getPostsByDESO - dunno why it doesn't work for all.. ,
addGlobalFeedBool: posts will contain a bool (but i kinda thought they already had an IsGlobal..?)
*/
try {
let batch = null;
let results = [];
let ct = 0;
let lastPostHash = params.postHashHex ? params.postHashHex : undefined;
while (ct < 300 && (batch === null || batch.length)) {
ct++;
batch = await this.getPosts({ pk: params.readerPk ? params.readerPk : undefined, postHashHex: lastPostHash, orderBy: params.orderBy ? params.orderBy : undefined, postContent: params.postContent ? params.postContent : undefined, numToFetch: params.numToFetch ? params.numToFetch : 50, mediaRequired: params.mediaRequired ? params.mediaRequired : false });
batch = batch.PostsFound;
if (batch.length) {
const filteredBatch = batch.filter(item => !blacklistedUsers.includes(item.PosterPublicKeyBase58Check));
lastPostHash = '';
if (filteredBatch.length) {
results.push(...filteredBatch);
}
lastPostHash = batch[batch.length - 1].PostHashHex;
}
}
if (fs) {
fs.writeFile('./src/scripts/dataStore/' + lastPostHash + '.json', JSON.stringify(results), (err) => {
if (err) { console.error(err); return; };
console.log("File has been created");
});
}
res = results;
} catch (error) {
console.log(error);
res = [];
}
break;
}
return res;
}
}
module.exports = DesoLite;and then with the way 11ty.js works, here’s my call to it
const DesoLite = require('../scripts/desolite.js');
module.exports = function() {
const desolite = new DesoLite({
pkey: '',
userName: 'kitty4d'
});
return desolite.getFrom('get-posts-all', {});
};it’s just that easy~